Determining a next value of a parameter for system simulation

ABSTRACT

An embodiment of an apparatus includes a simulator and a determiner. The a simulator is configured to simulate a system and to propagate at least one state of the simulated system through time in response to a value of a parameter, and the determiner is configured to determine a next value of the parameter in response to a characteristic of another state of the model and a representation of at least one level set.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is related to and claims the benefit of theearliest available effective filing date(s) from the following listedapplication(s) (the “Related Applications”) (e.g., claims earliestavailable priority dates for other than provisional patent applicationsor claims benefits under 35 USC §119(e) for provisional patentapplications, for any and all parent, grandparent, great-grandparent,etc. applications of the Related Application(s)). All subject matter ofthe Related Applications and of any and all parent, grandparent,great-grandparent, etc. applications of the Related Applications,including any priority claims, is incorporated herein by reference tothe extent such subject matter is not inconsistent herewith.

RELATED APPLICATIONS

For purposes of the USPTO extra-statutory requirements, the presentapplication constitutes a continuation-in-part of United States PatentApplication No. TBD [Attorney Docket No. 2914-002-03], entitledDETERMINING A NEXT VALUE OF A SYSTEM-SIMULATION PARAMETER IN RESPONSE TOA REPRESENTATION OF A PLOT HAVING THE PARAMETER AS A DIMENSION, namingMichael H. Baym, Philip Andrew Eckhoff, Daniel Jay MacDonald, Nathan P.Myhrvold, Karima R. Nigmatulina, Charles Whitmer, Lowell L. Wood, Jr. asinventors, filed Aug. 16, 2011, which is filed on the same date as theinstant application, or is an application of which a currentlyco-pending application is entitled to the benefit of the filing date;

For purposes of the USPTO extra-statutory requirements, the presentapplication constitutes a continuation-in-part of United States PatentApplication No. TBD [Attorney Docket No. 2914-003-03], entitledDETERMINING A NEXT VALUE OF A SYSTEM-SIMULATION PARAMETER IN RESPONSE TOREPRESENTATIONS OF PLOTS HAVING THE PARAMETER AS A DIMENSION, namingMichael H. Baym, Philip Andrew Eckhoff, Daniel Jay MacDonald, Nathan P.Myhrvold, Karima R. Nigmatulina, Charles Whitmer, Lowell L. Wood, Jr. asinventors, filed Aug. 16, 2011, which is filed on the same date as theinstant application, or is an application of which a currentlyco-pending application is entitled to the benefit of the filing date.

The United States Patent Office (USPTO) has published a notice to theeffect that the USPTO's computer programs require that patent applicantsreference both a serial number and indicate whether an application is acontinuation, continuation-in-part, or divisional of a parentapplication. Stephen G. Kunin, Benefit of Prior-Filed Application, USPTOOfficial Gazette Mar. 18, 2003. The present Applicant Entity(hereinafter “Applicant”) has provided above a specific reference to theapplication(s) from which priority is being claimed as recited bystatute. Applicant understands that the statute is unambiguous in itsspecific reference language and does not require either a serial numberor any characterization, such as “continuation” or“continuation-in-part,” for claiming priority to U.S. patentapplications. Notwithstanding the foregoing, Applicant understands thatthe USPTO's computer programs have certain data entry requirements, andhence Applicant has provided designation(s) of a relationship betweenthe present application and its parent application(s) as set forthabove, but expressly points out that such designation(s) are not to beconstrued in any way as any type of commentary and/or admission as towhether or not the present application contains any new matter inaddition to the matter of its parent application(s).

SUMMARY

The foregoing summary is illustrative only and is not intended to be inany way limiting. In addition to the illustrative aspects, embodiments,and features described above, further aspects, embodiments, and featureswill become apparent by reference to the drawings and the followingdetailed description.

An embodiment of an apparatus includes a simulator and a determiner. Thesimulator is configured to simulate a system and to propagate at leastone state of the simulated system through time in response to a value ofa parameter, and the determiner is configured to determine a next valueof the parameter in response to a characteristic of a state of thesimulated system and a representation of at least one level set.

As compared to existing simulation techniques, such a determiner may,for example, lead one to discover more quickly a range ofinput-parameter values that corresponds to a sought-after result, or todiscard more quickly a range of input-parameter values that does notcorrespond to a sought-after result. For example, when simulating adisease-transmission system, such a determiner may lead one to discovermore quickly a range of values for a disease-remediation parameter(e.g., the percentage of the population vaccinated) that corresponds toa sought-after probability of eliminating a disease from a region; or,such a determiner may lead one to discover more quickly a range ofvalues for the disease-remediation parameter that is insufficient toeliminate the disease from the region.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a block diagram of an embodiment of a system.

FIG. 2 is a block diagram of an apparatus/tool for simulating a system.

FIG. 3 is a timing diagram of the states of a system model as anembodiment of the apparatus of FIG. 2 propagates the states throughtime.

FIG. 4 is a plot of a probability that an individual will travel versusthe time since the individual last traveled.

FIG. 5 is a plot of a probability that at least one individual in aregion will travel by a particular time.

FIG. 6 is a block diagram of an embodiment of an apparatus/tool forsimulating a stochastic system.

FIG. 7 is a block diagram of an embodiment of the state tracker of FIG.6, where the state tracker tracks the states of a system model on aper-item basis.

FIG. 8 is a block diagram of an embodiment of the state tracker of FIG.6, where the state tracker tracks the states of a system model on acompartmentalized basis.

FIG. 9 is a plot of binomial distributions for a state characteristic.

FIG. 10 is a topological plot of a land mass.

FIG. 11 is a plan view of the plot of FIG. 10.

FIG. 12 is a plot of initial parameter points that are randomlygenerated by an embodiment of the determiner of FIG. 6.

FIG. 13 is a plot of initial parameter points that are systematicallygenerated by an embodiment of the determiner of FIG. 6.

FIG. 14 is a plot of a simulation-result surface generated by anembodiment of the plotter of FIG. 6, where the surface includes a peakthat extends to and above a state-characteristic threshold.

FIG. 15 is a top plan view of the plot of FIG. 14.

FIG. 16 is a plot of another simulation-result surface generated by anembodiment of the plotter of FIG. 6, where the surface includes peakthat does not extend to or above a state-characteristic threshold.

FIG. 17 is a plot of a simulation-result surface generated by anembodiment of the plotter of FIG. 6, where the surface includes a valleythat extends to and below a state-characteristic threshold.

FIG. 18 is a top plan view of the plot of FIG. 17.

FIG. 19 is a plot of another simulation-result surface generated by anembodiment of the plotter of FIG. 6, where the surface includes valleythat does not extend to or below a state-characteristic threshold.

FIG. 20 is a plot of a simulation-result surface generated by anembodiment of the plotter of FIG. 6, where the surface includes a slopethat exceeds a slope threshold.

FIG. 21 is a top plan view of the plot of FIG. 20.

FIG. 22 is a group of plots of simulation-result surfaces generated byan embodiment of the plotter of FIG. 6, where the plots each have atleast one common dimension.

FIG. 23 is a flow chart of an embodiment of a technique for determininga next value of a parameter for system simulation.

FIG. 24 is a flow chart of another embodiment of a technique fordetermining a next value of a parameter for system simulation.

FIG. 25 is a flow chart of another embodiment of a technique fordetermining a next value of a parameter for system simulation.

FIG. 26 is a flow chart of an embodiment of a technique for generatingan initial simulation-result surface.

FIG. 27 is a flow chart of an embodiment of a technique for using asimulation-result surface to determine a next value of a parameter forsystem simulation, for running a simulation of a system in response tothe next value of the parameter, and for updating the result surface.

DETAILED DESCRIPTION

In the following detailed description, reference is made to theaccompanying drawings, which form a part hereof. In the drawings,similar symbols typically identify similar components, unless contextdictates otherwise. The illustrative embodiments described in thedetailed description, drawings, and claims are not meant to be limiting.Other embodiments may be utilized, and other changes may be made,without departing from the spirit or scope of the subject matterpresented here.

One or more embodiments are described with reference to the drawings,wherein like reference numerals may be used to refer to like elementsthroughout. In the following description, for purposes of explanation,numerous specific details are set forth in order to provide a thoroughunderstanding of the one or more embodiments. It may be evident,however, that one or more embodiments may be practiced without thesespecific details. In other instances, well-known structures and devicesare shown in block diagram form in order to facilitate describing one ormore embodiments.

Organizations such as the World Health Organization (WHO), have, formany years, tried to eradicate certain human diseases such as malaria,polio, and guinea worm, from the planet. But to date, the only humandisease that is known to have been eradicated by human intervention issmallpox—rinderpest, an animal-specific disease, has also beeneradicated by human intervention.

Therefore, to assist these organizations in developing cost-effectivedisease-eradication or disease-elimination campaigns that have goodprobabilities of success, scientists have developed simulationapparatuses/tools (hereinafter simulation tools) for simulatingdisease-transmission systems. In general, the goal of adisease-eradication campaign is eradication of a disease from the entireplanet, and the goal of a disease-elimination campaign is eradication ofa disease from one or more regions of the planet; that is, eliminationof a disease from the entire planet may be called eradication. But“eradication” and “elimination” are used interchangeably herein to meanremoval of a disease from one or more regions of the planet, or from theentire planet.

Such a simulation tool typically includes a model of adisease-transmission system, where this system model includes one ormore complex microsimulations of phenomena that compose the system.Typically, such a microsimulation defines a relationship between aphenomenon to be simulated and the environment in which the phenomenonis immersed. For example, a microsimulation may define the life cycle ofa malaria parasite (there is more than one such parasite) while inside amosquito host in terms of the weather of the region in which themosquito resides. Or, a microsimulation may define the life cycle of amalaria parasite while inside a human host in terms of the immunitylevel of the host (e.g., is the host vaccinated, was the host previouslyinfected), or it may define the transmission of a malaria parasite fromhuman to mosquito and from mosquito to human in terms of the level ofinteraction between humans and mosquitoes (e.g., percentage of bed-netcoverage, percentage of sleeping quarters with screens) in a region ofinterest.

To determine whether or not a disease-elimination campaign is likely tobe successful, one first provides the simulation tool with values of theparameters (e.g., vaccination coverage, education, campaign duration) ofthe campaign.

Next, the simulation tool simulates the disease-transmission system inresponse to the input campaign parameters, and generates one or morecampaign output results, such as the probability that the number ofindividuals infected with the disease at the end of the campaign is zero(i.e., successful elimination).

Then, if the campaign results are satisfactory, then one may use thesimulated campaign-parameter values in an actual campaign knowing thatthese values are likely to yield a sought-after result (e.g., a 99% orhigher probability of elimination).

Alternatively, if the campaign results are not satisfactory, then onemay re-simulate the disease-transmission system with a different set ofcampaign-parameter values, and repeat this procedure until he/she findsa set of values that corresponds to a sought-after result, or untilhe/she determines that no such set of values is likely to exist.

Unfortunately, as discussed below in more detail, this hit-or-missprocedure of trying different sets of campaign-parameter values mayrequire a total simulation time that is relatively long, and that may beimpractical.

More specifically, an elimination-campaign simulation tool may simulatea disease-transmission system by simulating a large number of individualstochastic states/events at discrete times over the campaign period,where the probabilities of states/events occurring or not occurring maybe weighted according to parameterized data. For example, the simulationtool may, as part of a simulation, predict whether a particularin-silico (i.e., simulated) individual within a particular region willbecome infected with a malaria parasite at a particular time based on aprobability of infection that depends on factors such as the region'sweather, whether the individual has a bed net, whether the bed net istreated with a mosquitocide, and the probability that the individualwill use the bed net if he/she has one. The number of simulatedindividuals infected at any given simulation time is a state of thesimulated system, as is the number of individuals that are not infectedat any given simulation time. Furthermore, when an uninfected individualbecomes infected is an event of the simulated system, as is when aninfected individual becomes uninfected.

Due to the large number states/events and other complex phenomena thatcompose the simulated disease-transmission system, the time that asimulation tool requires to run a simulation may be significant,particularly when the tool is simulating a campaign on a very largescale (e.g., over a large geographical area, over a large population, orover a long campaign period). For example, just a single simulation at aparticular set of campaign-parameter values may take hours, days, oreven weeks depending on the tool's computing power, and one may need torun many (e.g., 100, 1000, or even more) simulations to determine withina reasonable statistical certainty whether the set of campaign-parametervalues corresponds to a sought-after result (e.g., a 99% or higherprobability of eradicating the disease).

Therefore, designers of tools for simulating disease-transmissionsystems and other complex systems (e.g., weather systems, economicsystems) continue to search for ways to reduce the overall time that asimulation tool requires to simulate a system sufficiently for aparticular purpose, such as for determining whether a particular set ofinput-parameter values corresponds to a sought-after result.

As discussed below, an embodiment of a simulation tool may decrease itsoverall simulation time by reducing the time that that the tool takes todiscover whether there exists a range of input-parameter values thatcorresponds to a sought-after result, and, if such a range exists, byreducing the time that the tool takes to characterize this range. Forexample, an embodiment of a simulation tool may select a next set ofinput-parameter values based on simulation results yielded by one ormore preceding sets of input-parameter values.

Before discussing such an embodiment of a simulation tool in moredetail, some general system-related concepts, andsystem-simulation-related concepts, are discussed.

FIG. 1 is a block diagram of a feedback system 10, which has one or morestates S and generates one or more output signals 12 in response to oneor more input signals 14 and one or more feedback signals 16. Asdiscussed below, depending on the system 10, the output, input, andfeedback signals 12, 14, and 16 may be electromagnetic or other types ofsignals, or may be quantities other than signals.

If the system 10 is a continuous-time system, then the values of itsstates S may vary smoothly with time; such a system is sometimes calledan analog system.

In contrast, if the system 10 is a discrete-time system, then the valuesof its states S change at discrete time intervals; such a system issometimes called a digital system.

The output signals 12 are the signals that constitute the values of oneor more of the states S at the output of the system 10. If the system 10is a continuous-time system, then the output signals 12 may be analogsignals, whereas if the system 10 is a discrete-time system, then theoutput signals may be digital signals having values or levels thatchange from discrete-time interval to discrete-time interval.Furthermore, there may be more states S than there are output signals12; that is, not every state need be an output of the system 10.

The input signals 14 may also be analog or digital signals, and are, orrepresent, stimuli in response to which the system 10 generates theoutput signals 12. In other words, the input signals 14 “drive” thesystem 10.

The feedback signals 16 are equal to, or are otherwise derived from, oneor more of the output signals 12. For example, a feedback signal 16 maybe related to a corresponding output signal 12 by a scalar (e.g., 5, ½),may be a linear (e.g., derivative, integral) or non-linear function ofthe corresponding output signal, or may be a combination of multipleoutput signals 12 or signals derived from such output signals.Furthermore, although shown as being derived from the output signals 12,one or more of the feedback signals 16 may be derived from intermediatestate values that the system 10 generates but does not provide as outputsignals. For example, if the system has multiple stages, then a feedbacksignal 16 may be derived from the output of one of the stages other thanthe last stage.

The system 10 may be linear system, which has a behavior that one maypredict almost exactly for virtually any set of initial conditions andvirtually any values of the input signals 14, or the system may be anon-linear system (sometimes called a chaotic system), which has abehavior that one may not predict exactly for virtually any set ofinitial conditions and virtually any values of the input signals. Asignificant number of predictable linear systems (e.g., a cruise-controlsystem for a vehicle, a temperature-control system for a building) aredesigned by humans, and a significant number of themore-difficult-to-predict non-linear systems are naturally occurringsystems (e.g., weather systems), or are human-built or human-influencedsystems over which humans may have only partial control (e.g., economicsystems, disease-transmission systems).

An example of a relatively simple linear system 10 is a cruise-controlsystem for maintaining the speed of a vehicle at a relatively constantvalue, where an output signal 12 is a signal that represents the actualspeed of the vehicle, an input signal 14 is a signal that represents thedesired speed of the vehicle, and a feedback signal 16 equals the outputsignal.

In operation, the cruise-control system 10 generates the output signal12 such that the difference between the input and feedback signals 14and 16 stably tends toward zero.

For example, suppose the input signal 14 indicates a desired speed of 60miles per hour (mph), and at an arbitrary initial time the output signal12 and feedback signal 16 indicate an actual speed of 50 mph. Inresponse to this initial condition, the cruise-control system 10increases the speed of the vehicle until the output and feedback signals12 and 16 indicate a speed of 60 mph, and thereafter maintains the speedof the vehicle at 60 mph.

Similarly, suppose the input signal 14 indicates a desired speed of 60miles per hour (mph), and at an arbitrary initial time the output signal12 and feedback signal 16 indicate an actual speed of 70 mph. Inresponse to this condition, the cruise-control system 10 decreases thespeed of the vehicle until the output and feedback signals 12 and 16indicate a speed of 60 mph, and thereafter maintains the speed of thevehicle at 60 mph.

Consequently, regardless of the initial speed of the vehicle and otherinitial conditions of the system, and regardless of subsequentconditions (e.g., different incline of road, different weather) of thesystem, the cruise-control system 10 exhibits the predictable (andsought-after) behavior of bringing the speed of the vehicle to 60 mph,and thereafter maintaining the speed at 60 mph.

An example of a non-linear system 10 is a human-disease-transmissionsystem, such as described above, that governs the spread of a diseasesuch as malaria or polio among humans in a given region. In such asystem, the input signals 14 represent the values of actual quantitiessuch as average rainfall and average temperature for the region, thepercentage of the region's population who are vaccinated, and the timingof the vaccinations, the output signals 12 represent the actual valuesof system states such as the number of individuals infected with adisease, the number of individuals immune to the disease, and thelocation of each individual, and the feedback signals 16 represent theoutput signals or a subset thereof.

In operation, unlike in a linear system, the output signals 12 may notconverge to predictable target values that are independent of theinitial conditions and the subsequent conditions of the system duringthe time period of interest; instead, the values of the output signals12 at the end of a time period of interest may depend heavily not onlyon the initial conditions of the system, but also on the subsequentconditions of the system that develop during the time period ofinterest.

For example, consider a disease-transmission system having input signalswith given values and where one of the output signals 12 represents thenumber of infected individuals at the end of adisease-elimination-campaign period of ten years. Even assuming givenvalues for the input signals 14, the number of infected individuals atthe end of the campaign period may depend heavily on the number ofindividuals infected at the beginning of the campaign period (an initialcondition), and may also depend heavily on the weather and the migrationpatterns of the individuals in the region of interest during thecampaign period (subsequent conditions).

Consequently, even assuming unchanging input signals 14, the number ofinfected individuals at the end of an elimination-campaign period mayvary significantly depending on the initial and subsequent conditions ofthe system.

FIG. 2 is a block diagram of an embodiment of a system-simulation tool20 for simulating a system such as the system 10 of FIG. 1. The tool 20includes a system simulator 22, which receives input data 24, inputparameters 26, and feedback signals 28, and which generates outputsignals 30 and output results 32 in response to a system model 34.

The simulator 22 may include a computing apparatus (e.g., a computingsystem having one or more processors) that operates under the control ofthe system model 34, which may include software, firmware, hardware, ora combination or sub-combination thereof, that causes the simulator toact, at least in an algorithmic sense, as the system being simulated.

The simulator 22 simulates a system by propagating the states of thesimulated system through time at discrete time steps. This may be trueeven if the system being simulated is a continuous-time system such as acruise-control system. Because there are conventional techniques forgenerating a discrete-time model of a continuous-time system, suchtechniques are not further described.

The input data 24 are signals that represent input information that anoperator of the tool 20 does not manipulate for purposes of thesimulation. For example, where the system being simulated is acruise-control system, such input data may include the weight of thevehicle and the power profile of the engine. Or, where the system beingsimulated is a disease-transmission system, the input data 24 mayinclude population and hours-of-daylight data for a region.

The input parameters 26 are signals that represent input informationthat an operator of the tool 20 may manipulate for purposes of thesimulation. For example, where the system being simulated is acruise-control system, such input information may include the terrain(e.g., flat, mountainous) over which a vehicle will travel during thesimulation. Or, where the simulated system is a disease-transmissionsystem, the input parameters 24 may include the percentage of thepopulation to receive a vaccination during the simulation period, andthe timing of these vaccinations.

The feedback signals 28 may be equal to, or derived from, the outputsignals 30 and output results 32.

The output signals 30 represent the values of one or more of the statesS of the simulated system after each discrete time step. For example, ifthe system being simulated is a cruise-control system, then one of theoutput signals 30 may represent the simulated speed of the simulatedvehicle. Or, if the system being simulated is a disease-transmissionsystem, then one of the output signals 30 may represent the number ofindividuals infected with a disease.

The output results 32 include information that may be derived from theoutput signals 30, and thus from one or more of the states S, of thesimulated system. For example, if the system being simulated is acruise-control system, then an output result 32 may be the time requiredfor the vehicle to re-attain the cruising speed set by an input signal24 in response to a sudden change in terrain (e.g., from flat road tosteep incline). Or, if the system being simulated is adisease-transmission system, then an output result 32 may be whether adisease is eradicated (i.e., the number of infected individuals is zero)at the end of a disease-elimination campaign period, or may be theprobability that the elimination-campaign eradicated the disease.

The system model 34 mathematically defines the system to be simulated.For example, if the system to be simulated is a linear system, then thesystem model 34 may include a set of linear equations thatmathematically define the behavior of the simulated system regardless ofthe system's configuration. Or, if the system to be simulated is anon-linear system, then, as further discussed below, the system model 34may define all the possible configurations of the simulated system, andmay define the respective behaviors of the simulated system for each ofthese possible system configurations. Alternatively, the system model 34may define the probabilities by which the system simulator 22 may selectsystem configurations for simulation, and may define the behaviors ofthe simulated system for each of these selected configurations.Furthermore, although the system model 34 is described in the singular,it may include multiple models or microsimulations that together formthe system model as described above.

FIG. 3 is a timing diagram of the x states S₁-S_(x-1) of the systemsimulated by the simulator 22 (FIG. 2) and defined by the system model34 (FIG. 2) at each of the z time steps t₀-t_(z-1) through which thesimulator propagates the states. The interval between each time step maybe constant such as in constant-time-interval simulation, or it may varyfrom step to step such as in event-based-time-interval simulation.Furthermore, the output signals 30, and thus the output results 34 (notshown in FIG. 3), may have different values at each time step.

The operation of the simulation tool 20 is now discussed in conjunctionwith FIGS. 2 and 3.

An operator of the simulation tool 20 first selects the input-datasignals 24 and the values of the input-data signals (e.g., from an inputdata base), and selects the input parameters 24 and the values of theinput parameters (e.g., enters them via a user interface). The operatormay also be able to select the feedback signals 28 that are fed back tothe input of the simulator 22.

Next, the simulator 22 simulates the system by effectively propagatingthe states S of the simulated system, as defined by the system model 34,through a period of time t₀-t_(z-1), discrete time step by discrete timestep.

Still referring to FIG. 2, modeling and simulation of a non-linearsystem is further discussed.

As discussed above in conjunction with FIG. 1, the output signals 12 ofa non-linear system 10 may depend heavily on the initial conditions andthe subsequent conditions of the system. For example, for adisease-transmission system, the number of infected individuals at theend of a disease-elimination campaign period may depend heavily on thenumber of individuals infected at the beginning of the campaign period(an initial condition), and may also depend heavily on the migrationpatterns of the individuals and the weather in the region of interestduring the campaign period (subsequent conditions).

One potential way to simulate a non-linear system is to assume aparticular configuration of the system, where the particularconfiguration includes known values of all initial conditions and of allsubsequent conditions at each time step t of the simulation period.

Then, by propagating the states S of the so-configured system throughtime in response to the model 34, the simulator 22 may yield the outputsignals 30 and output results 32 for this particular systemconfiguration.

For example, if the simulated system is a disease-transmission system,then one may assume a particular system configuration that includes aparticular number of infected individuals at the beginning of adisease-elimination campaign, and a particular respective location foreach individual at each time step of the simulation period.

But simulating a non-linear system 10, such as a disease-transmissionsystem, based on a single system configuration is akin to trying topredict the future. For example, it would be difficult to impossible topredict exactly where each individual will be at the beginning of acampaign period, and how each individual will travel and interact withother individuals during the campaign period. Therefore, it would bedifficult to impossible to predict what system configuration will existduring an actual campaign period.

Another potential way to simulate a non-linear system is for thesimulation tool 20 to determine all possible configurations of thesystem, and then to simulate the system for each of these possibleconfigurations. For example, if the non-linear system to be simulated isa disease-transmission system, then the simulation tool 20 would provideall possible outcomes of a disease-elimination campaign. And from theseoutcomes, the tool 20 may evaluate the level of success of the campaign.For example, the tool may calculate the probability of success as theratio of the number of simulated system configurations in which adisease is eradicated (i.e., zero infected individuals at the end of thecampaign period) to the total number of possible system configurations.

A problem with this simulate-all-configurations approach is that, atleast for a large non-linear system, it may be impractical or impossibleto determine each and every possible configuration of the system. Forexample, in a disease-transmission system that includes 100,000 people,and for a simulator 22 that operates with discrete time steps spacedapart by a constant interval equal to one day, it would be impracticalor impossible to determine each possible set of locations for these100,000 people each day over a campaign period such as 1-10 years.

And even if one could determine all possible configurations of a system,the time that the simulator 22 would need to simulate all of theseconfigurations may be too long to be practical.

To illustrate the complexity of determining all possible configurationsof a large non-linear system and simulating the system at each of theseconfigurations, consider the game of chess. A chess board has thirty twopieces (e.g., individuals) and sixty four squares (e.g., locations). Ithas been estimated that the number of possible sets of chess moves(e.g., configurations of the locations of individuals), where each setof moves constitutes a complete game of chess (e.g., adisease-elimination campaign period) is approximately 10¹²⁰. To put thisnumber in perspective, the total number of atoms in the observableuniverse has been estimated at “only” around 10⁸⁰! So one may see howexpanding the number of chess pieces to 100,000 and expanding the chessboard to many hundreds or thousands of square miles, increases thenumber of possible configurations to a number that makes theall-configurations approach impractical for all but the most simplenon-linear systems.

Another potential technique for simulating a non-linear system is todetermine a statistically independent subset of all possibleconfigurations of the system, to use the simulation tool 20 to simulateeach of the possible configurations in the subset, and to extrapolatethe simulation results for this subset of configurations to all possibleconfigurations. This technique is somewhat similar to forecastingelection results by polling. For example, one brute-force way toforecast a presidential election is to phone every registered voter inthe U.S. and to ask each voter for which candidate he/she will vote; butthis would require over 200 million phone calls! Therefore, pollstersphone a small, statistically independent subset of registered voters(e.g., 1000 voters who represent all major political demographics), andthen use statistics, and perhaps other mathematical techniques, toestimate the outcome of the election within a certain error bound.

One way to implement this statistical technique is to model a non-linearsystem as a stochastic (i.e., probabilistic) system, and to simulate thesystem by assuming that at each time step the configuration of thesystem is based on probabilities that items (e.g., individuals) of thesystem will act in a particular manner.

For example, one may assume that in a disease-transmission system, thelocations of the individuals in the system at any time step depends on arespective probability for each individual that the individual willtravel from on location to another location during the interval betweenthe current time step and the immediately prior time step.

FIG. 4 is a plot 40 of a distribution of the probability that anindividual will travel between the current and immediately prior timesteps versus the number of days since the individual last traveled,where the area under the probability curve 42 equals one. For example,according to the curve 42, the probability that the individual willtravel if it has been 30 days since he/she last traveled isapproximately 5%.

FIG. 5 is a plot 50 of a distribution of the probability that anindividual will travel by any particular day from the day he/she lasttraveled, where the curve 52 is the integral of the curve 42 of FIG. 4;in some applications, the plot 50 may be more useful for systemsimulation than the plot 40. For example, the plot of FIG. 5 indicatesthat there is an approximately 30% chance that an individual will travelto a different location within the 30 days following the day he lasttravelled.

Referring to FIGS. 2 and 5, the simulator 22 may simulate the locationsof individuals at any time step by simulating the migration of theindividuals according to the probability curve 52. For example, focusingon a single individual who last traveled thirty days prior to a relativetime step t₀, the simulator 22 generates a random number from 0-1, andfrom the random number determines whether the individual has traveled inthe interval between t₀ and t₁. Because at thirty days the probabilitycurve 52 indicates a probability of 30%, if the random number is lessthan or equal to 0.3, then the simulator 22 determines that theindividual has traveled in the interval between t₀ and t₁; but if therandom number is greater than 0.3, then the simulator 22 determines thatthe individual has not travelled in the interval between t₀ and t₁. Thesimulator 22 may simulate the movement of the remaining simulatedindividuals in a similar manner.

By simulating the movement of individuals and the actions of othersystem items in such a probabilistic manner, the simulator 22 simulatesthe system in one of the possible system configurations, where thesimulator selects the simulated system configuration by randomly drawingfrom probability-distribution curves such as the curve 52 for individualmigration. Such a technique is sometimes called a Monte Carlo technique,or a Monte Carlo based technique.

To simulate a statistically independent subset of all the possiblesystem configurations, the simulator 22 may repeat the above procedurefor a number of times (e.g., 100, 1,000, 10,000) that is sufficient toyield a statistically independent subset of system configurations. Thatis, the simulator 22 may “run” through the simulation with the sameinput-data and input-parameter values for a number of times that issufficient to yield simulation results from a statistically independentsubset of the possible system configurations.

The randomness involved in the simulator's selection of the systemconfigurations to be simulated helps to insure statistical independenceof the selected configurations, while the probability weighting of thisrandomness helps to insure that more weight is given to more likelyconfigurations than to less likely configurations. For example, if thesimulation period is ten years, it is very unlikely that none of the100,000 individuals in a disease-transmission system will travel fromone location to another during the simulation period. Therefore,although it is possible that the simulator 22 may randomly select such aconfiguration, the chances of this are much less likely than the chancesthat the simulator 22 will randomly select a configuration in which atleast some of the individuals travel during the simulation period.

Referring to FIG. 2, as described above, depending upon the size of thesimulated non-linear system and the computing power of the simulationtool 20, even a single run through a system simulation by the simulator22 may consume a significant amount of execution time (e.g., a fewminutes, a few hours, a few days, or even a few weeks).

And taking into account that the simulator 22 may execute multiple(e.g., 100, 1,000, 10,000) simulation runs to yield a statisticallysufficient set of output signals 30 and output results 32 for a singleset of input-data and input-parameter values, it may take a significantamount of time for one using the simulator to find a set of input-dataand input-parameter values that corresponds to a sought-after result(e.g., a greater than 99% probability of disease elimination), or todetermine that such a set of input values does not exist. For example,assume that the simulator 22 is simulating a disease-transmission systemto determine if there is a range of percentages of the number ofindividuals vaccinated that will lead to elimination of a disease, and,if such a range exists, to find this range. Also assume that but for theinput parameter 26 that represents the percentage of vaccinationcoverage, the values of the input-data signals 24 and the values of theother input-parameters 26 remain constant. To determine and identifysuch a range of the percentage of vaccination coverage, one may run thesimulator 22 with a pre-selected number of differentpercentage-of-vaccination values, for example, one hundred percentagevalues from 1% to 100% in steps of one percent. One may see which ofthese percentage values corresponds to the highest probability ofdisease elimination, and then he/she may investigate some fractionalpercentages around this highest-probability percentage value to identifythe boundaries of a range of percentage values that corresponds to asought-after probability of elimination. But at perhaps 10,000 or moresimulation runs per simulated percentage value, such a simulationprotocol may take a significant amount of time. And if there aremultiple significantly different percentages that yield a sought-afterresult, then one may decide to investigate fractional percentages aroundall of these different percentages to develop the boundaries of multipleranges of suitable vaccination-percentage values. Furthermore, considerthat value of more than one input parameter may be changeable (e.g.,vaccination percentage, vaccination timing, and bed-net coverage), thusgeometrically increasing the number of combinations of input-parametervalues that may be simulated. Consequently, using such a methodicaltechnique to hone in on a set of input-parameter values that correspondsto a sought-after result, or to determine that no such set ofinput-parameter values exists, may take an impractically long time.

FIG. 6 is a block diagram of an embodiment of a simulation tool 60 that,compared to a conventional simulation tool, may decrease the timerequired to identify a set of input-parameter values that corresponds toa sought-after result, or may decrease the time required to determinethat no such set of input-parameter values exists.

The simulation tool 60 includes an input database 62, anext-parameter-value determiner 64, a system simulator 66, a systemmodel 68, a state-characteristic calculator 70, a plotter 72, aninterface 74, and an output database 76. The simulator tool 60 may beinstalled on, or may include, a computing apparatus that is configuredby software, firmware, hardware, or a combination or sub-combination ofsoftware, firmware, and hardware, to perform the functions of thesimulation tool, including the functions of the input database 62,determiner 64, simulator 66, system model 68, calculator 70, plotter 72,interface 74, and output database 76. These and other components of thesimulation tool 60 may be separately identifiable software or firmwaremodules or circuits, or they may be logical components of modules orcircuits that perform the functions of more than one of thesecomponents. For example, the simulation tool 60 may include means forperforming the functions of these and other tool components, and thesemeans may be may be separately identifiable software or firmware modulesor circuits, or they may be modules or circuits that perform thefunctions of more than one of these tool components.

The input database 62 stores input data regarding the system to besimulated, the conditions of the environment in which the system isdisposed, and other quantities that may influence the system. Forexample, the input database 62 may store, e.g., topographicalinformation, weather information, and sunrise/sunset information, for ageographical region. In an embodiment, the input database 62 may storedata that an operator of the tool 60 does not wish to manipulate as aninput parameter, e.g., because such data (e.g., topographicalinformation) is fixed and not manipulatable in the real world.

The next-parameter-value determiner 64 determines a next value of one ormore input parameters in a manner that, compared to a conventionalsimulation tool, allows the simulation tool 60 to hone in more quicklyon a set of parameter values that corresponds to a sought-after result,or to determine more quickly that no such set of parameter valuesexists; for example, the determiner may allow the simulation tool tohone in on a set of parameter values, or to determine that no such setof values exists, with fewer simulation runs than a conventionalsimulation tool. The determiner 64 may determine such next-parametervalues in response to one or more state-characteristic values from thecalculator 70, and from the representations of one or moreresult-surface plots and one or more level sets (described below) fromthe plotter 72. The operation of the determiner 64 is described in moredetail below in conjunction with FIGS. 10-27.

The system simulator 66 includes a state tracker 78, and, referring toFIG. 3, propagates the x states S₀-S_(x-1) of the simulated systemthrough time in z discrete steps t₀-t_(z-1) in response to one or moreinput-data values and one or more input-parameter values from thedeterminer 64, and one or more output-state values fed back to the inputof the simulator. The simulator 66 propagates the states S through timeby determining the respective value that each state S has at each timestep t in response to the system model 68. For example, for thesimulation of a malaria-transmission system, the model 68 may defineinfluences that environmental conditions or other quantities have on thenumber of infected individuals (a state of the simulated system) at atime t; examples of such conditions and quantities include the number ofinfected individuals at the previous time step t−1, the number ofinfected mosquitoes at t−1, the temperature and rainfall at t−1, thebed-net coverage at t−1, the percentage of individuals vaccinated att−1, and the migration of individuals during the interval between t−1 tot. And from these defined influences and the values of thesesystem-influencing conditions and quantities, the simulator 66 maycalculate the number of infected individuals at time t; for example, thehigher the temperature and rainfall at t−1, the more infectedmosquitoes, and thus the more infected individuals, at t. The simulator66 may calculate the values of the other states of the simulated systemin a similar manner. Furthermore, the intervals between the steps t maybe uniform, or they may vary if event-based timing is used. Moreover,although not shown in FIG. 3, the simulator 66 may propagate some of thestates S through time in uniform time steps, and may, at leasteffectively, propagate other states S through time in event-basednon-uniform time steps. The operation of the system simulator 66 isdiscussed in more detail below in conjunction with FIGS. 23-27 accordingto an embodiment.

The state tracker 78 keeps track of the current value of each state S ofthe simulated system. For example, if a state S is the number ofindividuals infected, then the state tracker 78 stores this number,which the simulator 66 updates at each time step. The state tracker isfurther described below in conjunction with FIGS. 7 and 8 according toan embodiment.

The system model 68 may include one or more microsimulations, and mayinclude software, firmware, hardware, or a combination orsub-combination of software, firmware, and hardware, that defines asystem to be simulated by the simulator 66. For example, for amalaria-transmission system, the system model 68 may stochasticallydefine the system states, and may define system phenomena that affectthe system states such as the migration patterns of individuals, thelife cycle of a malaria parasite within a human host, the life cycle ofa malaria parasite within a mosquito host, the life cycle of a mosquito,the interaction between individuals and mosquitoes, and the infection ofindividuals by mosquitoes and of mosquitoes by individuals. The systemmodel 68 may also stochastically define dependencies of such systemphenomena on environmental conditions and other system-influencingquantities such as temperature, rainfall, humidity, the number ofinfected individuals at a time step, and the number of infectedmosquitoes at a time step.

The calculator 70 calculates one or more characteristics of, or one ormore characteristics that are otherwise related to, one or more states Sof the simulated system. For example, after a number of runs of thesimulator 66 at a particular set of input-parameter and input-datavalues for a simulated disease-elimination campaign, the calculator 70may calculate the probability that the simulated campaign eradicated thedisease, i.e., the probability that at the end of the campaign period,the number of infected individuals (and for malaria, the number ofinfected mosquitoes) is zero. The calculator 70 may also calculate thestatistical variance of, or the statistical uncertainty in, thisprobability. The calculator 70 may make its calculation using anysuitable function such as a Bayesian prior, a beta function orincomplete beta function, which are subsets of a Bayesian prior, or abinomial distribution, which is a subset of an incomplete beta function.The operation of the calculator 70 using a binomial distribution isfurther described below in conjunction with FIG. 9 according to anembodiment.

The plotter 72 generates representations of one or more plots based onone or more of the following: the values of one or more statecharacteristics from the calculator 70, one or more input-parametervalues from the determiner 64 and the interface 74, and one or morestate values output from the simulator 66—“representation” indicatesthat the plotter 72 need not generate an actual visual plot (although itmay), but that it may instead generate a representation of a plot in,e.g., computer memory, which the simulator tool 60 includes or to whichthe simulator tool otherwise has access. Alternatively, the plotter 72may receive the input-parameter values from one, not both, of thedeterminer 64 and interface 74. For example, the plotter 72 may generatea representation of an N-dimensional result surface having respectiveinput parameters as N−1 of these dimensions, and having a statecharacteristic as the remaining dimension. As a more detailed example,assume that for a malaria-transmission system, an operator of the tool60 is investigating the relationship among vaccination coverage (aninput parameter), bed-net coverage (another input parameter), and theprobability of eradicating malaria (a state characteristic). The plotter72 may generate a representation of an N=3-dimensional result surfacehaving vaccination coverage as a first dimension, bed-net coverage as asecond dimension, and probability of elimination as a third dimension.The plotter 72 may also generate a representation of one or more levelsets for this result surface, and the determiner 64 may use such aresult surface and level set to determine a respective next value of thevaccination-coverage parameter, the bed-net use parameter, or of both ofthese parameters. Furthermore, the plotter 72 may generate other typesof plots, such as a topographical plot of a region showing a timeprogression of the distribution of infected individuals within theregion. Operation of the plotter 72, including the generation of resultsurfaces and level sets, is further described below in conjunction withFIGS. 10-22 according to an embodiment.

The interface 74 interfaces other components of the simulation tool 60to each other, and includes a graphical user interface (GUI) 80, whichallows an operator to configure and to use the simulation tool.

The interface 74 may also store in the output database 76 a “snap shot”of the values of the states S of the simulated system at each time stept of the simulation period. These snap shots may allow an operator tosee the development of one or more states of the simulated system as thesimulator 66 propagates these states through time, and may facilitate anoperator's step-by-step analysis of a simulation run or an operator'sstep-by-step comparison of multiple simulation runs.

Furthermore, the interface 74 may store in the output database 76 foreach simulation run the state characteristics generated by thecalculator 70 and the representations of the spatial points and resultsurfaces generated by the plotter 72.

Moreover, the interface 74 may store in the output database 76 for eachsimulation run the set of input-data and input-parameter values providedto the simulator 66 for the simulation run. Because this set of inputvalues, or a subset of this set of input values, may be the coordinatesof a representation of a spatial point that is generated by the plotter72, this set of input values is sometimes referred to as the “simulationpoint” of the simulation run. And an operator of the simulation tool 60may be described as using the tool to find a simulation point thatcorresponds to a sought-after result. Furthermore, the interface 74 mayprovide this set of input values to the determiner 64 for a nextsimulation run that may use the same set, or a slightly modified set, ofinput values. Providing the set of input values from a prior simulationrun to the determiner 64 may eliminate the need to re-enter, manually orotherwise, all of the input-data values from the input database 64 andall of the unmodified input-parameter values via the GUI 80.

In addition, the interface 74 and GUI 80 may allow a user to configure,control, or otherwise interact with the simulation tool 60.

Furthermore, the interface 74 may allow an operator to select input-dataand input-parameter values to be input to the simulator 66. For example,if an operator would like to simulate a malaria-transmission system inMadagascar, then he/she may use the interface 74 to configure thesimulation tool 62 such that Madagascar-relevant input data (e.g.,weather-related data, population data) from the input database 62 isprovided to the simulator 66.

Moreover, an operator may select via the interface 74 input parametersto be used in a simulation, and may also select values of these inputparameters. For example, for simulating a malaria-transmission system,an operator may select vaccination coverage and bed-net coverage asinput parameters, and may select initial values for these coverages.

In addition, an operator may configure the system model 68 via theinterface 74. For example, there is more than one malaria parasite, andone parasite may behave differently under given conditions than anotherparasite. Therefore, an operator may configure the model 68 for aparticular malaria parasite. Or, if individuals of one type (e.g., childvs. adult) react, on average, differently to a malaria parasite thanindividuals of another type, then an operator may provide the model 68with the percentages of each type of individual in a region of interest(alternatively, these percentages may come from the input database 62).

Furthermore, an operator may configure the determiner 64 via theinterface 74. For example, an operator may specify the parameter orparameters for which the determiner 64 will determine a next value aftera set of simulation runs, and may also specify an algorithm or othercriteria that the determiner uses to determine such next value(s). Anexample of such an algorithm is described below in conjunction withFIGS. 10-22 according to an embodiment.

Moreover, an operator may, via the interface 74, specify one or morestate characteristics and configure the calculator 70 to generate thevalues of the specified one or more characteristics.

In addition, an operator may, via the interface 74, specify one or moreplots and configure the plotter 72 to generate representations of thespecified one or more plots.

Furthermore, an operator may configure, via the interface 74, the GUI 80to display tool-generated quantities such as one or more state valuesfrom the simulator 66, one or more state-characteristic values from thecalculator 70, or one or more plots from the plotter 72.

The output database 76 may store the state-value snapshots from thesimulator 66, the state-characteristic values from the calculator 70,the representations of the points and plots from the plotter 72, and thesets of input-data and input-parameter values input to the simulator asdescribed above, and may also store other data.

Still referring to FIG. 6, alternate embodiments of the simulation tool60 are contemplated. For example, the system model 68 may be part of thesystem simulator 66.

FIG. 7 is a block diagram of an embodiment of the state tracker 78 (FIG.6) for an embodiment of the system simulator 66 (FIG. 6) that tracksstates S of a simulated system on a per-item basis, i.e., by the itemsthat have one or more of the tracked states.

As discussed above in conjunction with FIG. 6, the simulated systemincludes items (e.g., individuals, mosquitoes) each having one or morestates S (e.g., infected, immune, vaccinated, item location).

The state tracker 78 saves the values of these states as the simulator66 propagates these states through time, and updates the values of thesestates at each time step t.

To track states on a per-item basis, the state tracker 78 includes oneset of memory locations for each item in the simulated system, where thenumber of memory locations in the set is the number of states S that theitem may have; not all items need have the same number of states, andnot all memory locations need be the same size. Each of such memorylocations may be disposed in a respective state-memory register.Therefore, if there are p items included in the simulated system, thenthe state tracker 78 includes the following number of state registers:

Number of state registers=[x=the number of states S foritem_(—)0]+[y=the number of states S for item_(—)1]+ . . . +[z=thenumber of states S for item_(—) p−1])  (1)

For example, assume that the system being simulated is amalaria-transmission system, item_0 is an individual of the simulatedsystem, item_1 is a mosquito of the simulated system, each individualhas the states “infected (yes/no)”, “vaccinated (yes/no)”, “location(identity of location)”, and “recovered (yes/no),” and each mosquito hasthe states “infected (yes/no)” and “able to infect a human (yes/no).”

In this example, the state tracker 78 may include for item_0 and everyother simulated individual three one-bit state registers for the states“infected,” “vaccinated,” and “recovered,” and a multi-bit stateregister for “location” (assuming that there are more than two possiblelocations), and may include for item_1 and every other simulatedmosquito two one-bit state registers for the states “infected” and “ableto infect a human.”

As discussed above in conjunction with FIG. 6, in an embodiment, at eachtime step t the simulator 66 updates the values of the states S storedin the registers of state tracker 78, and the interface 74 stores asnapshot of these state values in the output database 76 by loading thecontents of the state-tracker registers into the output database.

Still referring to FIG. 7, alternate embodiments of the per-item statetracker 78 are contemplated. For example, instead of including a stateregister for each state of each simulated item, the tracker 78 mayinclude one state register per item, where each state register includesa respective register location for each state of the item.

FIG. 8 is a block diagram of an embodiment of the state tracker 78 ofFIG. 6 for an embodiment of the system simulator 66 (FIG. 6) that tracksthe states S of a simulated system on a compartmentalized basis.

As discussed above in conjunction with FIGS. 6-7, the system simulator66 simulates a system that includes items (e.g., individuals,mosquitoes) each having one or more states S (e.g., “infected,”“immune,” “vaccinated,” “location”).

The state tracker 78 saves the values of these states as the simulator66 propagates these states through time, and updates the values of thesestates at each time step t.

To track these states on a compartmentalized basis, the state tracker 78includes one set of counters for each type of item modeled, where thenumber of counters in the set equals the number of possible combinationsof state values that an item of the corresponding type may have; not allitem types need have the same number of possible combinations of statevalues, and not all counters need be the same size. Each of suchcounters stores the number of items having the corresponding state-valuecombination, and may be configured as a conventional counter or as arespective memory register. Therefore, if there are q item typesincluded in the system model, then the state tracker 78 includes thefollowing number of state-value-combination counters:

Number of state-value-combination counters=[w=the number of possiblestate-value combinations for item_type_(—)0]+[v=the number of possiblestate-value combinations for item_type_(—)1]+ . . . +[u=the number ofpossible state-value combinations for item_type_(—) q−1]  (2)

For example, suppose that the system being simulated is amalaria-transmission system, item_type_0 includes simulated individuals,item_type_1 includes simulated mosquitoes, each individual has thestates “infected (yes/no)”, “vaccinated (yes/no)”, “location (identityof one of three locations)”, and “previously infected (yes/no),” andeach mosquito has the states “infected (yes/no)” and “able to infect ahuman (yes/no),” where the possible values for each state are within theparentheses.

In this example, for each item in item_type_0, i.e., for each simulatedindividual, there are three states each having two possible values andone state having three possible values; therefore, each individual mayhave one of 2·2·2·3=24 possible state-value combinations at each timestep t, and the state tracker 78 includes twenty four state-valuecounters for item_type_0. And, at each time step t, each counter storesthe number of individuals having the corresponding state-valuecombination. For example, one possible state-value combination that anindividual may have is “infected=yes”, “vaccinated=yes”,“location=Antananarivo, Madagascar”, and “previously infected=yes;”therefore, the counter corresponding to this state-value combinationupdates and stores the number of individuals that have this combinationof state values at each time step t.

Similarly, for each item in item_type_1, i.e., for each modeledmosquito, there are two states each having two possible values;therefore, each mosquito may have one of 2·2=4 possible state-valuecombinations at each time step t, and the state tracker 78 includes fourstate-value counters for item_type_1. And at each time step t, eachcounter updates and stores the number of mosquitoes having thecorresponding state-value combination. For example, one possiblestate-value combination that a mosquito may have is “infected=yes” and“able to infect a human=no;” therefore, the item_type_1 countercorresponding to this state-value combination updates and stores thenumber of mosquitoes that have this combination of state values at eachtime step t.

At each time step t, after the simulator 66 updates the values of thecounters, the interface 74 stores a snapshot of these counter values inthe output database 76 by loading the contents of the counters into theoutput database as described above in conjunction with FIG. 6.

Still referring to FIG. 8, alternate embodiments of thecompartmentalized state tracker 78 are contemplated. For example,instead of including one set of counters for each item type, the statetracker 78 may include one set of counters for each item type in eachgeographic location (this would allow one to omit location as a statefor that item type) or other grouping. Furthermore, where the values ofsome states are mutually exclusive, this may reduce the number ofpossible combinations of state values, and thus may allow one to reduceor “prune” the number of counters in the state tracker 78. For example,for a disease-transmission system, if a state of a simulated individualstate is “infected,” and another state is “recovered,” then both ofthese states for an individual may not simultaneously have a value of“yes”; therefore, there may be no need for the state tracker 78 toinclude a counter for any state-value combination where the values ofboth of these states is “yes”.

Referring to FIGS. 7-8, in an embodiment, whether to implement aper-item state tracker 78 or a compartmentalized state tracker maydepend on the system being simulated. For example, if the simulatedsystem includes a relatively small number of items, then a per-itemstate tracker 78 may occupy less memory than a compartmentalized statetracker; conversely, if the simulated system includes a relatively largenumber of items, then a compartmentalized state tracker 78 may occupyless memory than a per-item state tracker.

FIG. 9 is a plot 90 of a cumulative distribution function that, in anembodiment, yields binomial-distribution curves 92-98, which thecalculator 70 of FIG. 6 may generate and may use to calculate astate-characteristic value in response to one or more correspondingstate values from the simulator 66, where the state-characteristic valuerepresents a probability.

The cumulative distribution is the dimension along the vertical axis ofthe plot 90, and the state characteristic is the dimension along thehorizontal axis. For example, the state characteristic may be theprobability of disease-elimination by the end of an elimination-campaignperiod, where the corresponding state is the number of infectedindividuals at the end of the campaign period.

In an example where the state characteristic is the probability ofsuccessful elimination, the calculator 70 generates a representation ofthe curve 92 from the number of successful eliminations and the numberof unsuccessful eliminations over a number of simulation runs at a setof input-data and input-parameter values (i.e., at a simulation point).The curve 92 has a mean equal to r, which is the value of the statecharacteristic, and, has a width w₉₂ between, for example, its 5% height100 and its 95% height 102. The plotter 72 may generate a representationof the curve 92 suitable for display via the GUI 80 (FIG. 6).

The curve 94, a representation of which the calculator 70 generates fromthe numbers of elimination successes and failures over a greater numberof simulation runs than from which it generates the representation ofthe curve 92, also has a mean equal to r and has a width w₉₄ between its5% height and 95% height.

Similarly, the curve 96, a representation of which the calculator 70generates from the numbers of elimination successes and failures over agreater number of simulation runs than from which it generates therepresentation of the curve 94, also has a mean equal to r, and has awidth w₉₆; and the curve 98, a representation of which the calculatorgenerates from the numbers of elimination successes and failures over agreater number of simulation runs than from which it generates therepresentation of the curve 96, also has a mean equal to r, and has awidth w₉₈.

Consequently, the more simulation runs on which a binomial-distributioncurve is based, the narrower its width w.

Furthermore, a binomial-distribution curve's width w is proportional tothe statistical uncertainty in, the standard deviation σ of, and thestatistical variance σ² of, the value r of the state characteristic;therefore, in general, the greater the number of simulation runs onwhich a binomial-distribution curve is based, the lower the statisticaluncertainty in, and the smaller the standard deviation σ and thevariance σ² of, r. Statistical uncertainty is dependent on the standarddeviation and the variance, and may be expressed as a function of one orboth of the standard deviation and variance. For example, if r=99.5%probability of success, the standard deviation σ=√0.15%, and thevariance σ²=0.15%, then the statistical uncertainty in r may beexpressed as ±√0.15% if the statistical uncertainty is expressed as afunction of σ, or may be expressed as ±0.15% if the statisticaluncertainty is expressed as a function of σ².

Therefore, an operator of the simulation tool 60 may configure thesystem simulator 66 to generate a number of simulation runs per eachsimulation point sufficient to yield the state-characteristic value rhaving a statistical uncertainty that is less than an uncertaintythreshold, and having a standard deviation σ that is less than astandard-deviation threshold or having a variance σ² that is less than avariance threshold.

Referring to FIGS. 6 and 9, operation of an embodiment of the calculator70 is described for an example where the modeled system is adisease-transmission system and the state characteristic is theprobability of successful disease elimination at the end of theelimination-campaign period at a single simulation point.

First, the simulator 66 runs a number, e.g., 100, 1,000, 10,000, ofsimulations of the modeled system at the simulation point. A result thateach of these simulations yields is a respective number of infectedindividuals at the end of the elimination campaign, where the respectivenumber either equals zero (successful elimination) or does not equalzero (unsuccessful elimination campaign).

Next, the calculator 70 labels as successes all simulation runs yieldinga number of infected individuals equal to 0, and labels as failures allsimulation runs yielding this number not equal to zero. For example, thenumber of successes may be represented by a and the number of failuresmay be represented by b such that a+b equals the number of simulationsruns.

Then, from a and b, the calculator 70 calculates the probability ofsuccess=mean r=a/(a+b), and generates a representation of abinomial-distribution curve, such as one of the curves 92-98, accordingto one or more known equations. Furthermore, an operator may configurethe tool 60 such that plotter 72 and GUI 80 cause the curve to bedisplayed on a display device that may be part of, or coupled to, theGUI.

Next, from the representation of the binomial-distribution curve, thecalculator 70 calculates the statistical uncertainty in, and thestandard deviation σ or variance σ² of, the probability of success r.

Then, the calculator 70 compares the statistical uncertainty andstandard deviation or variance to respective thresholds that a user mayhave entered via the GUI 80. If either the uncertainty or standarddeviation/variance is greater than its corresponding threshold, then thecalculator 70 may instruct the simulator 66 to run additionalsimulations at the corresponding simulation point in an effort to reducethe uncertainty and/or standard deviation/variance below thecorresponding thresholds, or may instruct the plotter 72 to ignore thissimulation point when generating or updating a result surface(generating and updating a result surface are described below).Alternatively, if the uncertainty and standard deviation/variance areless than or equal to the corresponding thresholds, then the calculator70 may pass the probability of success and the identity of thesimulation point to the plotter 72 for use in generating or updating aresult surface as described below.

Still referring to FIGS. 6 and 9, in an embodiment, the calculator 70may calculate one or more categories of uncertainty other thanstatistical uncertainty, and may represent these other categories asrespective state characteristics, or may generate one or more statecharacteristics that each represent a combination of two or more ofthese uncertainty categories. Conventional algorithms may be used forgenerating values for other categories of uncertainty such as thosedescribed below, and for combining these values. Furthermore, thecalculator 70 may compare one or more of these uncertainty values to oneor more respective thresholds, and may instruct the simulator 66 to runadditional simulations at the corresponding simulation point in aneffort to reduce these one or more uncertainties below the correspondingthresholds, or may instruct the plotter 72 to ignore this simulationpoint when generating or updating a result surface. Alternatively, ifthe one or more uncertainty values are less than or equal to thecorresponding thresholds, then the calculator 70 may pass theprobability of success and the identity of the simulation point to theplotter 72 for use in generating or updating a result surface.

Stochastic uncertainty is a measure of the uncertainty in a particularsimulation result. For example, suppose that the result is the number ofindividuals who are infected at the end of an elimination campaign. Ifall of the simulations yield numbers of remaining infected individualsbetween 0-10, then the stochastic uncertainty that the number ofremaining infected individuals is no more than 10 is relatively low. Butthe stochastic uncertainty that the number of remaining infectedindividuals is no more than five may depend on the probabilitydistribution of the number of remaining infected individuals between 1and 10. The stochastic uncertainty may be independent of the probabilityof success. For example, if all simulation runs yield a number ofremaining infected individuals being either 1 or 2, then the stochasticuncertainty in the number of remaining infected individuals isrelatively low, but the current estimate of the probability of successin eradicating the disease is zero because no simulations yield zeroremaining infected individuals. The calculator 70 may conventionallycalculate a value of stochastic uncertainty.

Parameter uncertainty is a measure of the uncertainty of a simulationresult based on the uncertainty in input data or in an input parameter.For example, it may not be accurately known in what percentage ofindividuals a vaccine is effective in inducing immunity to a disease. Soeven if the vaccination coverage (i.e., the percentage of individualsvaccinated) is accurately known, there may be uncertainty in the numberof individuals that the vaccination makes immune. If a simulation resultsuch as the number of individuals infected at the end of anelimination-campaign period is sensitive to the number of individualsmade immune by vaccination, then the uncertainty in the effectiveness ofthe vaccine may impart a significant uncertainty to the calculated valueof the probability of successful elimination of the disease. But anoperator may use the simulation tool 60 to simulate and account for thepossible effect of parameter uncertainty on a simulation result. Forexample, he/she may simulate a wide range of vaccination coverage underthe assumption that all vaccinations induce immunity, and see to whatrange of immunity percentages the disease elimination is sensitive. Forexample, if vaccination-induced immunity of 60% or greater correspondsto a sought-after probability of success that is relatively insensitiveto small changes in the value of vaccination-induced immunity, and it isknown that conservatively, the vaccine induces immunity in 90% ofindividuals, then vaccination coverage of about 67% or greatercorresponds to the sought-after probability of successful eliminationeven if 10% of the vaccinated individuals do not become immune. Usingthe simulation tool 60 to simulate the possible effects of parameteruncertainty on a simulation result is further described below inconjunction with FIGS. 20-21 according to an embodiment. Furthermore,the calculator 70 may calculate a value of parameter uncertainty.

Coverage uncertainty is the uncertainty in whether a particularintervention will be utilized. For example, if a malaria-eliminationcampaign calls for issuing bed nets to all children under 10 years old,then the uncertainty is a measure of how many of these children willactually receive a bed net, and how many of the children receiving a bednet will actually use it. A value for coverage uncertainty may bedetermined empirically from actual data (e.g., distribute bed nets tochildren and see how may children receive and use them), and an operatormay effectively input this value to the simulation tool 60 (FIG. 6) sothat the simulator 66 may account for it in simulations. For example,he/she may simulate a wide range of bed-net coverage under theassumption that all distributed bed nets will be used, and see to whatbed-net coverage percentages disease elimination is sensitive. Forexample, if bed-net coverage of 50% or greater corresponds to asought-after probability of successful elimination that is relativelyinsensitive to small changes in the value of bed-net coverage, and it isknown that conservatively, 90% of children to whom a bed net isdistributed will receive and use the bed net, then bed-net coverage ofabout 56% or greater corresponds to the sought-after probability ofsuccessful elimination even if 10% of the children to whom a bed net isdistributed do not receive or use it. Using the simulation tool 60 tosimulate the possible effects of coverage uncertainty on a simulationresult is further described below in conjunction with FIGS. 20-21according to an embodiment. Furthermore, the calculator 70 may calculatea value of coverage uncertainty.

Model-assumption uncertainty is the uncertainty in a simulation outcomecaused by an uncertainty in an assumption built into the system model 68(FIG. 6). Model-assumption uncertainty is similar to parameter andcoverage uncertainty, but it may be difficult to account for insimulations without the ability to modify the system model. For example,for a malaria-transmission system, the model 68 may assume that amosquito must live at least ten days before it is able to infect ahuman. But a simulation result such as probability of successfulelimination of malaria may be sensitive to a variation in the minimumnumber of days that a mosquito must live before being able to infect ahuman. Without the ability to change this number of days in the model,an operator may be unable to investigate the sensitivity of theprobability of successful elimination to this number of days asdescribed above for parameter and coverage uncertainty. Therefore, themodel 68 may be designed to allow an operator of the simulation tool 60to vary one or more model assumptions so that he/she may determine a setof input-parameter values that correspond to a sought-after result evenif a model assumption is at a worst-case value. Furthermore, thecalculator 70 may calculate a value of model-assumption uncertainty.

Still referring to FIG. 9, alternate embodiments of the calculator 70are contemplated. For example, the calculator 70 may pass thestate-characteristic value r and corresponding simulation point to theplotter 72 regardless of the uncertainty in, or standarddeviation/variance of, r. Furthermore, because a cumulative distributionfunction is a type of beta function, which is a type of Bayesian prior,instead of calculating a cumulative distribution function of astate-characteristic-probability value r to yield abinomial-distribution curve, the calculator 70 may calculate arepresentation of another type of beta function or Bayesian prior of r.Moreover, although described as calculating a probability, thecalculator 70 may calculate a state-characteristic value other than aprobability, and may do so using a function other than a cumulativedistribution function. In addition, instead of comparing the one or moreuncertainties in, and the standard deviation/variance of, r torespective thresholds, the calculator 70 may compare only one or none ofthe one or more uncertainties and standard deviation/variance to arespective threshold; furthermore, the calculator may analyze r in amanner that does not even involve uncertainty, standard deviation, orvariance.

FIG. 10 is a plot 110 of a three-dimensional topographical map 111,which is used to explain an embodiment of a level set. The x and y axesrespectively represent longitude and latitude, and the z axis representsaltitude (z=0 is sea level).

FIG. 11 is a plan view of the plot 110 of FIG. 10 in the x-y plane.

Referring to FIGS. 10 and 11, the map 111 is of a landmass 113 havingtwo mountains 114 and 115.

Suppose one would like an indication of the following: 1) the portions(if any) of the landmass 113 having an altitude greater than 10,000feet, 2) the portions (if any) of the landmass having an altitudegreater than 8,000 feet, 3) the portions (if any) of the landmass havingan altitude between 8,000 feet and 10,000 feet, and 4) the portions (ifany) of the landmass having an altitude between 8,000 and 10,000 feetwith a slope in the z dimension of at least 25°.

Referring to FIG. 10, one way to generate such an indication is toinsert into the map 111 an imaginary plane 116 at z=8,000 feet andanother imaginary plane 118 at z=10,000.

Referring to FIG. 11, the dashed curves 120 and 122 represent theintersection of the 8,000-foot plane 116 with the landmass 111 at themountains 114 and 115, the solid curves 124 and 126 represent theintersection of the 10,000-foot plane 120 with the landmass at themountains, and the curves 127 and 128 (which include portions of thecurves 120 and 124, and 122 and 126, respectively) represent theportions of the mountains having slopes of at least 25° between the8,000- and 10,000-foot planes.

Consequently, the portions of the landmass 113 within the curves 124 and126 are at altitudes greater than 10,000 feet, the portions of thelandmass within the curves 120 and 122 are at altitudes greater than8,000 feet, the portions of the landmass between the pairs of curves120, 124 and 122, 126 are at altitudes between 8,000 and 10,000 feet,and the portions of the landmass within the curves 127 and 128 are ataltitudes between 8,000 and 10,000 feet and have slopes of 25° or more.

Therefore, the points of the landmass 113 that form the curves 124 and126 may be collectively referred to as a 10,000-foot “level set;” thatis, the 10,000-foot level set is the set of points of the landmass atthe 10,000-foot level. Furthermore, the points of the landmass 113within, i.e., bounded by, the curves 124 and 126 may be referred to as a10,000-foot “level-set region;” that is, the 10,000-foot level-setregion is the set of points of the landmass above the 10,000-foot level.

Similarly, the points of the landmass 113 that form the curves 120 and122 may be collectively referred to as an 8,000-foot level set, thepoints of the landmass between the curve pairs 120,124 and 122,126 maybe referred to as an 8,000-10,000-foot level-set region, the curves 127and 128 may be referred to as a 8,000-10,000-foot-with-25°-slopeboundary, and the points of the landmass within the curves 127 and 128may be referred to as the 8,000-10,000-foot-with-at-least-25°-sloperegion. Furthermore, notice that, for example, the 10,000-foot level-setregion is within the 8,000 level-set region such that one level-setregion may include another level-set region, and a level set may boundmore than one level-set region.

Moreover, the collection of all level sets that may appear as part ofthe topographical map 111, or that one may otherwise use in conjunctionwith the topographical map, may be referred to as a “separatrix” of themap. For example, the altitude curves or contours that appear on atopographical map collectively form an altitude separatrix of the map,and the depth curves or contours that appear on a depth chart of a bodyof water collectively form a depth separatrix of the chart. Technically,a topographical map may include an infinite number of level sets,although only a small subset of these level sets may be identified orotherwise used.

Referring to FIGS. 12-22, embodiments of the following are described: atechnique for generating a representation of an initial version of aresult surface, examples of result surfaces, a technique for generatinga representation of a level set relative to a result surface, atechnique for selecting a next value of one or more input parameters inresponse to representations of a result surface and a level set, and atechnique for updating a representation of a result surface and arepresentation of a level set.

A result surface is an N-dimensional surface (N may be greater than,equal to, or less than two) having P dimensions that are respectiveinput parameters and having SC dimensions that are state-characteristicssuch that N=P+SC.

For example, where the simulation tool 60 (FIG. 6) is simulating amalaria-transmission system, the plotter 72 may generate arepresentation of a result surface having the following N=3 dimensions:vaccination coverage, bed-net coverage (both input parameters), andprobability of successful elimination of malaria at the end of thecampaign period (state characteristic).

FIG. 12 is a plan view of a plot 130 of simulation points havingcoordinates that the determiner 64 (FIG. 6) may randomly generate, andfrom which the plotter 72 (FIG. 6) may generate a representation of aninitial version of an N=3-dimensional result surface.

For example purposes, the plot 130 is described as having a firstdimension P₁ corresponding to malaria-vaccination coverage, a seconddimension P₂ corresponding to bed-net coverage, and a third dimension SC(normal to the P₁-P₂ plane) corresponding to the probability ofsuccessful malaria elimination, where each of these dimensions haspossible values ranging from 0% to 100%.

To reduce the processing time and complexity involved in generating andupdating a representation of a result surface, the plot 130 may notinclude all of the dimensions of a simulation point. For example, asimulation point may include, in addition to P₁, P₂, and SC, otherdimensions that correspond to input-data values and input-parametervalues that the simulation tool 60 (FIG. 6) holds constant during asimulation; examples of such held-constant input values include, e.g.,the average temperature and the average rain fall for a region. Butbecause the values of these other dimensions are held constant for thesimulation, the plotter 72 may omit these dimensions from therepresentation of the plot 130, e.g., in an effort to simplify theplotter's generating and updating of the representation of the resultsurface.

Therefore, if the plot 130 includes fewer dimensions than a simulationpoint, then one may refer to the points of the plot as “partial”simulation points, and may refer to the result surface, a representationof which the plotter 72 (FIG. 6) generates from these partial simulationpoints, as a “partial” result surface.

Conversely, if the plot 130 includes all of the dimensions of asimulation point, then one may refer to the points of the plot as “full”simulation points, and may refer to the result surface, a representationof which the plotter 72 (FIG. 6) generates from these full simulationpoints, as a “full” result surface.

Referring to FIGS. 6 and 12, in operation, an embodiment of thedeterminer 64 receives from the interface 74 input-data values andinput-parameter values for any input parameters other than vaccinationcoverage and bed-net coverage, where these input-data and parametervalues may have been previously entered or otherwise selected by anoperator via the GUI 80 as discussed above in conjunction with FIG. 6.

The determiner 64 also randomly generates a respective value for each ofthe vaccination coverage (dimension P₁) and bed-net coverage (dimensionP₂).

Next, the determiner 64 provides the input-data and input-parametervalues (including the randomly generated values for vaccination andbed-net coverage) forming the simulation point to the system simulator66.

Then, the simulator 66 runs, at this simulation point, a number (e.g.,100, 1,000, 10,000) of simulations sufficient for the calculator 70 toyield a probability of success (dimension SC) having one or moreuncertainties and standard deviation/variance that are each less than arespective threshold.

Next, the plotter 72 receives the values of the vaccination coverage andbed-net coverage (i.e., dimensions P₁ and P₂) from the determiner 64,receives the value of the probability of successful elimination (i.e.,dimension SC) from the calculator 70, and generates a representation ofthe plot 130 including a point 132 at coordinates corresponding to thereceived values of P₁, P₂, and SC.

Then, the determiner 64 receives from the interface 74 the sameinput-data values and the same other input-parameter values (i.e., thevalues of all input parameters except vaccination coverage and bed-netcoverage) that the simulator 66 used for the last simulation run (i.e.,the simulation tool 60 holds these input values constant), and randomlygenerates a respective next value for each of the vaccination coverage(dimension P₁) and bed-net coverage (dimension P₂).

Next, the determiner 64 provides the input-data and input-parametervalues forming a next simulation point to the simulator 66, which runs,at this next simulation point, a number of simulations sufficient forthe calculator 70 to yield a probability of success (dimension SC)having one or more uncertainties and standard deviation/variance thatare each less than a respective threshold.

Then, the plotter 72 updates the representation of the plot 130 toinclude a next point 134 in addition to the previous point 132, wherethe point 134 has coordinates corresponding to the received next valuesof P₁, P₂, and SC.

The determiner 64, simulator 66, calculator 70, and plotter 72 repeatthe sequence described in the preceding three paragraphs until therepresentation of the plot 130 includes a number of randomly generatedpoints sufficient for the plotter 72 to generate a representation of aninitial version of a result surface (not shown in FIG. 12) having thedimensions P₁, P₂, and SC. In FIG. 12, the random positioning of thepoints of the plot 130 in the P₁-P₂ plane results from the determiner 64randomly generating the values for vaccination coverage (dimension P₁)and bed-net coverage (dimension P₂).

The plotter 72 may generate the representation of the initial version ofthe result surface according to a number of techniques. For example, theplotter 72 may use a conventional curve-fitting algorithm to generate,as the representation of the initial version of the result surface, arepresentation of a surface that best fits the points of the plot 130.Furthermore, the number of randomly generated points that is sufficientfor the plotter 72 to generate the representation of the initial resultsurface may depend, e.g., on the surface-generating technique used. Forexample, a number of points ≧N, e.g., 10, 100, 1,000, or 10,000 may besufficient for the plotter 72 to generate a representation of an initialresult surface.

Examples of result surfaces are described below in conjunction of withFIGS. 14-22.

FIG. 13 is a plan view of a plot 140 of simulation points that havecoordinates that the determiner 64 (FIG. 6) may methodically generate,and from which the plotter 72 (FIG. 6) may generate a representation ofan initial version of an N=3-dimensional result surface. In anembodiment, the determiner 64 methodically generates the points of theplot 140 according to a grid pattern.

Referring to FIGS. 6 and 13, assuming that the plot 140 has the samedimensions as the plot 130 of FIG. 12 (i.e., P₁=vaccination coverage,P₂=bed-net coverage, SC=probability of successfully eradicatingmalaria), the determiner 64, calculator 70, and plotter 72 operate asdescribed above in conjunction with FIG. 12 such that the plottergenerates a representation of an initial version of an N=3-dimensionalresult surface, except that instead of randomly selecting the values forvaccination coverage and bed-net coverage, the determiner methodicallyselects these values according to a grid pattern. For example, thedeterminer 64 may select values of vaccination coverage and bed-netcoverage that yield points in the P₁-P₂ plane at the following (P₁, P₂)coordinates: (20%, 20%), (20%, 40%), (20%, 60%), . . . , (40%, 20%),(40%, 40%), (40%, 60%), . . . , (100%, 100%)—the determiner may selectthese pairs of P₁, P₂ values in any order. These points form afive-point-by-five-point grid, where adjacent points are equidistantfrom one another in the P₁ and P₂ dimensions.

Referring to FIGS. 12 and 13, alternate embodiments of a technique forgenerating a representation of an initial result surface arecontemplated. For example, the determiner 64 (FIG. 6) may generate theinput-parameter values according to techniques other than the random andmethodical techniques described above in conjunction with FIGS. 12 and13, respectively. Furthermore, the plotter 72 may generate arepresentation of an initial result surface having fewer or more thanN=3 dimensions. Moreover, an initial result surface may include asdimensions input-data quantities and input parameters that remainconstant throughout the simulation. In addition, regardless of thenumber N of dimensions of a result surface, the determiner 64 maygenerate values for fewer or more than two of these N dimensions.

FIG. 14 is a plot 150 of an N=3-dimensional result surface 152, whichincludes a peak 154; as described above, the plotter 72 (FIG. 6) maygenerate a representation of the result surface 152, and the interface74 (FIG. 6) may convert this representation into a form suitable fordisplaying the plot 150 of the result surface via the GUI 80 (FIG. 6).

FIG. 15 is a plan view of the plot 150, result surface 152, a level-setregion 156, and a level set 158.

As described below, the determiner 64 may determine a next value of aninput parameter in response to the peak 154 and the level set 158 whenan operator of the simulation tool 60 (FIG. 6) is searching for a valueof the input parameter that corresponds to a sought-after result, suchas a sought-after state characteristic, that is higher than a particularthreshold represented by the level set 158.

For example, assume that the tool 60 (FIG. 6) is simulating amalaria-transmission system, the dimensions of the result surface 152are P₁=vaccination coverage, P₂=bed-net coverage, and SC=probability ofsuccessfully eliminating malaria from a region, and an operator of thesimulation tool 60 is desirous of discovering what, if any, values of P₁and P₂ yield a value of SC greater than or equal to 99.0%.

Therefore, the plotter 72 (FIG. 6) determines if there is a level set at99.0% that includes at least one point of the result surface 152.Conceptually, the plotter 72 may generate a representation of animaginary plane 160 at SC=99.0% that is parallel with the P₁-P₂ plane,and any portions of the result surface 152 that are level with thisplane form the 99.0% level set.

In this example, there are portions of the peak 154 of the resultsurface 152 that are level with and higher than the imaginary plane 160,and these portions respectively form a 99.0% level set 158 and a 99%level-set region 156, where the level set 158 is the intersection of theimaginary plane 160 with the result surface 152.

Referring to FIGS. 6, 14, and 15, operation of an embodiment of thedeterminer 64, simulator 66, calculator 70, and plotter 72 is describedin conjunction with the result surface 152 and level set 158.

In an effort to further define the portions of the result surface 150that form the level set 158 and the level-set region 156, the determiner64 selects next values of vaccination coverage P₁ and bed-net coverageP₂ for simulation in response to the level set 158. For example, thedeterminer 64 may randomly select a point 162 of the plot 150 that iswithin a particular radial distance R from a reference point 164 of theresult surface 152 within the level-set region 156, and the P₁ and P₂coordinates of this randomly selected point 162 are the next values ofvaccination coverage and bed-net coverage, respectively. The particularvalues for the radial distance R and the reference point 164 of theresult surface 152 from which the radial distance R is measured may beselected by an operator of the tool 60 via the GUI 80. For example, theradial distance R may be √{square root over (N_(t))} and the referencepoint 164 of the result surface 152 from which the radial distance R ismeasured may be the center of mass of the portion of the result surfacewithin the level-set region 156. It is noted that neither the randomlyselected point 162 nor the reference point 164 need lie on the resultsurface 152, although one or both points may lie on the result surface.Where the randomly selected point 162 lies on the result surface 152, asimulation using the P₁ and P₂ coordinates of the randomly selectedpoint is typically not a “wasted” situation, because it is unlikely thatthe simulator 66 has simulated the P₁ and P₂ coordinates of the randomlyselected point before. This is because, as described above, the plotter72 may generate a representation of the result surface 152 from arelatively small number of points (as compared to the infinite number ofpoints that form the result surface); therefore, there may be a goodchance that the randomly selected point 162 is not one of the pointsfrom which the plotter previously generated the representation of thecurrent result surface.

Consequently, instead of haphazardly selecting next values of thevaccination coverage and bed-net coverage, the determiner 64 selects thenext values for these parameters in a methodical manner that may reducethe overall number of simulation points that the tool 60 needs tosimulate before finding a range of values of the vaccination coverageand bed-net coverage that corresponds to the sought-after result of a99.0% chance of successful malaria elimination. That is, by selectingthe next values of vaccination coverage and bed-net coverage from nearthe level set 158, the determiner 64 may avoid a “wild goose chase” ofnext values that are unlikely to yield a probability of eradicatingmalaria that is 99.0% or better.

Still referring to FIGS. 6, 14, and 15, the determiner 64 then passesthese next values of vaccination coverage and bed-net coverage, alongwith the values of any other input data and input parameters that definea full simulation point, to the simulator 66, which runs a number ofsimulations sufficient to allow the calculator 70 to yield a value ofthe probability of success having one or more uncertainties and standarddeviation/variance within respective thresholds, for example, asdescribed above in conjunction with FIG. 9.

Next, the plotter 72 generates a representation of a next point 166 ofthe plot 150 having coordinates (P₁, P₂, SC) respectively equal to thenext values of the vaccination coverage and bed-net coverage selected bythe determiner 64, and to the next value of the probability of successcalculated by the calculator 70. Consequently, although the next point166 has the same P₁ and P₂ coordinates as the point 162, it almostcertainly will have a different SC coordinate than the point 162.Furthermore, it is noted that the point 162 is not actually part of theplot 150, but is shown in FIGS. 14 and 15 for purposes of explanation;therefore, the plotter 72 does not use the point 162 to update theresult surface 152.

Then, the interface 74 saves the representation of the next plot point166 of the plot 150 in the output database 76; that is, the interfacesaves all of the input-data and input-parameter values, i.e., the fullsimulation point, that yielded the next plot point 166, and also storesthe SC value of this plot point. Note that the coordinates of the fullsimulation point are the same as N−1 coordinates of the next plot point166, but the point 166 has an additional coordinate SC.

Next, the determiner 64 uses the level set 158 to select next values ofthe vaccination coverage (P₁) and bed-net coverage (P₂), and thesimulator 66, calculator 70, and plotter 72 repeat their respectiveabove-described operations, which result in a representation of a nextpoint 168 of the plot 150.

The determiner 64, simulator 66, calculator 70, and plotter 72 continueto repeat their above-described operations until the plotter hasgenerated a number of next plot points. An operator of the simulationtool 60 may select this number, which may be greater than or equal toone, via the interface 74.

Then, the plotter 72 updates the representation of the result surface152 using the previously calculated points of the plot 150 (the pointsthat the plotter used to generate the representation of the pre-updatedversion of the result surface) in combination with the next plot pointsjust calculated. For example, the plotter 72 may use a curve-fittingalgorithm to determine the updated result surface as a surface that bestfits this combination of plot points. Therefore, because the plotter 72uses all previously generated points of the plot 150 to update therepresentation of the result surface 152, the result surface typicallybecomes a more accurate indicator of the simulation results as thenumber of simulation points that the simulator 66 simulates increases.The plotter 72 may also update the location of the reference point 164in response to the updated result surface.

Next, the plotter 72 updates the 99.0% level-set region 156 and levelset 158 based on the representation of the updated result surface152—because the result surface is updated, it may have a different shapethan shown in FIGS. 14 and 15.

Then, the determiner 64, simulator 66, calculator 70, and plotter 72repeat the above procedure until the iterations of the result surface152 have sufficiently converged to a final result surface that issufficient to allow an operator to select the values of vaccinationcoverage and bed-net coverage that he/she feels are the best values foractual implementation. For example, an operator may visually monitor thechanges from updated result surface to updated result surface via theGUI 80, and instruct, via the interface 74, the determiner 64, simulator66, calculator 70, and plotter 72 to repeat the above procedure untilhe/she feels that the result surface as sufficiently converged toward afinal form. Or, an operator may configure the simulation tool 60, viathe interface 74, to monitor the changes from updated result surface toupdated result surface (e.g., to monitor the entropy of the resultsurface), and to halt the above procedure when the change from updatedresult surface to updated result surface is below an operator-selectedsurface-change threshold.

After obtaining a final result surface 152, or a representation thereof,from the simulation tool 60, an operator may select the values of thevaccination coverage and the bed-net coverage to be used in an actualmalaria-elimination campaign from a point of the plot 150 that is on oroff the final result surface 152. For example, an operator may select asthe values of vaccination coverage and bed-net coverage the respectiveP₁ and P₂ coordinates of a peak point 170 within the level-set region156 because this peak point has the highest value of the SC coordinate,and thus has the highest probability of successful eradication. Or, anoperator may select as the values of vaccination coverage and bed-netcoverage the respective P₁ and P₂ coordinates of a point that is not onthe result surface 152, but that is near to a point of the resultsurface within the level-set region 156 or on the level set 158.

Still referring to FIGS. 6, 14, and 15, alternate embodiments of theabove-described next-value-determination-and-result-surface-updatingtechnique are contemplated. For example, although described inconjunction with the simulation of a malaria-transmission system, anembodiment of the technique may be applicable to the simulation of otherdisease-transmission systems and of non-disease-transmission systems.Furthermore, although described in conjunction with a single level setand single level-set region, an embodiment of the technique iscompatible with the use of multiple level sets, multiple level-setregions, or both multiple level-sets and multiple level-set regions.Moreover, an embodiment of the technique is compatible with inputparameters other than vaccination coverage and bed-net coverage, andwith one or more state characteristics other than probability ofsuccessful malaria elimination. In addition, although the referencepoint 164 is described as being within the level set 158 in the P₁ andP₂ dimensions, the reference point may be located outside of the levelset in the P₁ and P₂ dimensions.

FIG. 16 is a plot 180 of an N=3-dimensional result surface 182, whichincludes a peak 184; as described above, the plotter 72 (FIG. 6) maygenerate a representation of the result surface, and the interface 74(FIG. 6) may convert this representation into a form suitable fordisplaying the plot of the result surface via the GUI 80 (FIG. 6).

Referring to FIGS. 6 and 16, in an example that is similar to thatdescribed above in conjunction with FIGS. 14-15, assume that thesimulation tool 60 is simulating a malaria-transmission system, thedimensions of the result surface 182 are P₁=vaccination coverage,P₂=bed-net coverage, and SC=probability of successfully eliminatingmalaria from a region, and an operator of the simulation tool isdesirous of discovering what, if any, values of P₁ and P₂ yield a valueof SC greater than or equal to 99.0%.

Therefore, the plotter 72 (FIG. 6) determines if there is a level set at99.0% that includes at least one point of the result surface 182.Conceptually, the plotter 72 may generate a representation of animaginary plane 186 at SC=99.0% that is parallel with the P₁-P₂ plane,and any portions of the result surface 182 that are level with thisplane form the 99.0% level set.

But in this example, unlike the example described above in conjunctionwith FIGS. 14-15, there are no portions of the peak 184 of the resultsurface 182 that are level with or higher than the imaginary plane 186;therefore, there is no 99.0% level set.

Referring to FIGS. 6 and 16, operation of an embodiment of thedeterminer 64, simulator 66, calculator 70, and plotter 72 is describedin conjunction with the result surface 182.

In an effort to determine whether a 99.0% level set exists, thedeterminer 64 selects next values of vaccination coverage P₁ and bed-netcoverage P₂ for simulation in response to the peak 184. For example, thedeterminer 64 may randomly select a point 187 of the plot 180 that iswithin a particular radial distance R from a reference point 188 of thepeak 184, and the P₁ and P₂ coordinates of this randomly selected point187 are the next values of vaccination coverage and bed-net coverage,respectively. The particular values for the radial distance R and thereference point 188 of the result surface 182 from which the radialdistance R is measured may be selected by an operator of the simulationtool 60 via the GUI 80. For example, the radial distance R may be√{square root over (N_(t))} and the reference point 188 of the resultsurface 182 may be the center of mass of the peak 184. It is noted thatneither the randomly selected point 187 nor the reference point 188 needlie on the result surface 182, although one or both points may lie onthe result surface. Where the randomly selected point 187 lies on theresult surface 182, a simulation using the P₁ and P₂ coordinates of therandomly selected point is typically not a “wasted” situation forreasons similar to those described above in conjunction with FIGS.14-15.

Consequently, instead of haphazardly selecting next values of thevaccination coverage and bed-net coverage, the determiner 64 selects thenext values for these parameters in a methodical manner that may reducethe overall number of points that the tool 60 needs to simulate in orderto determine whether there exist values of the vaccination coverage andbed-net coverage that yield the sought-after result of at least a 99.0%chance of successful malaria elimination. That is, by selecting the nextvalues of vaccination coverage and bed-net coverage from nearby the peak184, the determiner 64 may avoid a “wild goose chase” of next valuesthat are unlikely to yield information as to whether there exist nextvalues that yield a probability of eradicating malaria that is 99.0% orbetter.

Still referring to FIGS. 6 and 16, the determiner 64 then passes to thesimulator 66 these next values of vaccination coverage and bed-netcoverage, along with the values of any other input data and other inputparameters that, together with the next values of vaccination andbed-net coverage, define a full simulation point.

Then, the simulator 66 runs a number of simulations sufficient to allowthe calculator 70 to yield a value of the probability of success havingone or more uncertainties and standard deviation/variance withinrespective thresholds as described above in conjunction with FIG. 9.

Next, the plotter 72 generates a representation of a next point 190 ofthe plot 180 having coordinates (P₁, P₂, SC) respectively equal to thenext values of the vaccination coverage and bed-net coverage selected bythe determiner 64, and to the next value of the probability of successcalculated by the calculator 70. Although the next point 190 and point187 have the same P₁ and P₂ coordinates, they almost certainly havedifferent SC coordinates. Furthermore, the point 187 is not a point ofthe plot 180 that the plotter 72 will use to update the result surface182, but is shown in FIG. 16 for purposes of explanation.

Then, the interface 74 saves the representation of the next point 190 ofthe plot 180 in the output database 76; that is, the interface saves thefull coordinates (including input-data and other input-parameter valuesthat are not dimensions of the plot 180) of the next point 190 in theoutput database.

Next, the determiner 64 selects next values of the vaccination coverage(P₁) and bed-net coverage (P₂) nearby the peak 184, and the simulator66, calculator 70, and plotter 72 repeat their respectiveabove-described operations, which result in the plotter generating arepresentation of a next point 192 of the plot 180.

The determiner 64, simulator 66, calculator 70, and plotter 72 continueto repeat their above-described operations until the plotter hasgenerated a number of next plot points. An operator of the simulationtool 60 may select this number, which may be greater than or equal toone, via the interface 74.

Then, the plotter 72 generates a representation of an updated resultsurface 182 using the previously calculated points of the plot 180 (thepoints that the plotter used to generate the pre-updated version of theresult surface) in combination with the next plot points justcalculated. For example, the plotter 72 may use a curve-fittingalgorithm to determine the representation of the updated result surfaceas a representation of a surface that best fits this combination of plotpoints. And the plotter 72 may also update the location of the referencepoint 188.

Next, the plotter 72 evaluates the representation of the updated resultsurface 182 and determines whether a 99.0% level set exists—because theresult surface is updated, it may have a different shape than shown inFIG. 16.

If no 99.0% level set exists, then, the determiner 64, simulator 66,calculator 70, and plotter 72 repeat the above procedure until theiterations of the result surface 182 result in a 99% level set, or havesufficiently converged to a final result surface that is sufficient toindicate that no 99% level set is likely to exist. For example, anoperator may visually monitor the changes from updated result surface toupdated result surface via the GUI 80, and instruct, via the interface74, the determiner 64, simulator 66, calculator 70, and plotter 72 torepeat the above procedure until a 99% level set exists, or until he/shefeels that the result surface as sufficiently converged toward a finalform sufficient to indicate that a 99% level set is unlikely to exist.Or, an operator may configure the interface 74 to monitor the changesfrom updated result surface to updated result surface (e.g., to monitorthe entropy of the result surface), and to halt the above procedure wheneither a 99% level set is discovered or the change from updated resultsurface to updated result surface is below an operator-selectedthreshold.

If a 99% level set is found, then the determiner 64, simulator 66,calculator 70, and plotter 72 may revert to an embodiment of theprocedure described above in conjunction with FIGS. 14-15.

If no 99% level set is found, then an operator may “tweak” one or morevalues of the input data and input parameters, other than vaccinationcoverage and bed-net coverage, that form the full simulation point, oran operator may take other action such as using the simulation tool 60to simulate interventions other than vaccination coverage and bed-netcoverage.

Still referring to FIGS. 6 and 16, alternate embodiments of theabove-described next-value-determination-and-result-surface-updatingtechnique where no sought-after level set exists (at least initially)are contemplated. For example, alternate embodiments described above inconjunction with FIGS. 14-15 may also be compatible with an embodimentdescribed in conjunction with FIG. 16. Furthermore, although describedas searching for the existence of a single level set, an embodiment ofthe technique may search for multiple level sets.

FIG. 17 is a plot 200 of an N=3-dimensional result surface 202, whichincludes a valley 204; as described above, the plotter 72 (FIG. 6) maygenerate a representation of the result surface, and the interface 74(FIG. 6) may convert this representation into a form suitable fordisplaying the plot of the result surface via the GUI 80 (FIG. 6).

FIG. 18 is a plan view of the plot 200, result surface 202, valley 204,a level-set region 206, and a level set 208.

Referring to FIGS. 6, 17, and 18, as described below, the determiner 64may determine a next value of an input parameter in response to thevalley 204 and the level set 208 when an operator of the simulation tool60 is searching for a value of the input parameter that corresponds to asought-after result, such as a sought-after state characteristic, thatis lower than a particular threshold represented by the level set 208.

For example, assume that the simulation tool 60 is simulating amalaria-transmission system, the dimensions of the result surface 202are P₁=vaccination coverage, P₂=bed-net coverage, and SC=the averagecost of providing the vaccination coverage and bed-net coverage, and anoperator of the simulation tool is desirous of discovering what, if any,values of P₁ and P₂ yield a value of SC less than US $100,000,000 ($100M). In this example, the system model 68 (FIG. 6) may define a stateS₁=cumulative cost of the vaccination coverage over the campaign periodand a state S₂=cumulative cost of the bed-net coverage over the campaignperiod; therefore, SC equals the average of S₁+S₂ over all simulationruns at a particular simulation point.

Consequently, the plotter 72 determines whether there is a level set at$100 M that includes at least one point of the result surface 202.Conceptually, the plotter 72 may generate a representation of animaginary plane 210 at SC=$100 M that is parallel with the P₁-P₂ plane,and any portions of the result surface 202 that are level with thisplane form the $100 M level set.

In this example, there are portions of the valley 204 of the resultsurface 202 that are level with and lower than the imaginary plane 210,and these portions respectively form the $100 M level set 208 and the$100 M level-set region 206, where the level set 208 is the intersectionof the imaginary plane 210 with the result surface 202.

Referring to FIGS. 6, 17, and 18, operation of an embodiment of thedeterminer 64, simulator 66, calculator 70, and plotter 72 is describedin conjunction with the result surface 202, level-set region 206, andlevel set 208.

In an effort to further define the portions of the result surface 200within the level-set region 206 and that form the level set 208, thedeterminer 64 selects next values of vaccination coverage P₁ and bed-netcoverage P₂ for simulation in response to the level set 208. Forexample, the determiner 64 may randomly select a point 212 of the plot200 that is within a particular radial distance R from a reference point214 of the result surface 202 within the level-set region 206, and theP₁ and P₂ coordinates of this randomly selected point 212 are the nextvalues of vaccination coverage and bed-net coverage, respectively. Theparticular values for the radial distance R and the reference point 214of the result surface 202 from which the radial distance R is measuredmay be selected by an operator of the tool 60 via the GUI 80. Forexample, the radial distance R may be √{square root over (N_(t))} andthe reference point 214 may be the center of mass of the portion of theresult surface within the level-set region 206. It is noted that neitherthe randomly selected point 212 nor the reference point 214 need lie onthe result surface 202, although one or both of these points may lie onthe result surface. Where the randomly selected point 212 lies on theresult surface 202, a simulation using the P₁ and P₂ coordinates of therandomly selected point is typically not a “wasted” situation forreasons similar to those described above in conjunction with FIGS.14-15.

Consequently, instead of haphazardly selecting next values of thevaccination coverage and bed-net coverage, the determiner 64 selects thenext values for these parameters in a methodical manner that may reducethe overall number of simulation points that the tool 60 needs tosimulate before finding values of the vaccination coverage and bed-netcoverage that correspond to the sought-after result of a cost of $100 Mor less for vaccinations and bed nets. That is, by selecting the nextvalues of vaccination coverage and bed-net coverage from within thelevel-set region 206 or nearby the level-set region or level set 208,the determiner 64 may avoid a “wild goose chase” of next values that areunlikely to yield a cost of no more than $100 M for vaccinations and bednets.

Still referring to FIGS. 6, 17, and 18, the determiner 64 then passesthese next values of vaccination coverage and bed-net coverage, alongwith the values of any input data and any other input parameters thatdefine a full simulation point, to the simulator 66, which runs a numberof simulations sufficient to allow the calculator 70 to yield a value ofthe vaccination-and-bed-net-coverage cost having one or moreuncertainties and standard deviation/variance within respectivethresholds, for example, as described above in conjunction with FIG. 9.

Next, the plotter 72 generates a representation of a next point 216 ofthe plot 200 having coordinates (P₁, P₂, SC) respectively equal to thenext values of the vaccination coverage and bed-net coverage selected bythe determiner 64, and to the next value of the vaccination and bed-netcost calculated by the calculator 70. Although the points 212 and 216have the same P₁ and P₂ coordinates, they almost certainly have adifferent SC coordinate. Furthermore, the point 212 is not a point thatthe plotter will use to update the result surface 202, but this point isshown in the plot 200 for example purposes.

Then, the interface 74 saves the representation of the next point 216 ofthe plot 200 in the output database 76; that is, the interface saves inthe output database all of the input-data and input-parameter valuesthat yielded the next point 216, as well as the SC coordinate of thisnext point.

Next, the determiner 64 uses the level set 208 to select next values ofthe vaccination coverage (P₁) and bed-net coverage (P₂), and thesimulator 66, calculator 70, and plotter 72 repeat their respectiveabove-described operations, which results in the plotter generating arepresentation of a next point 218 of the plot 200.

The determiner 64, simulator 66, calculator 70, and plotter 72 continueto repeat their above-described operations until the plotter hasgenerated a number of next plot points. An operator of the tool 60 mayselect this number, which may be greater than or equal to one, via theinterface 74.

Then, the plotter 72 updates the result surface 202 using the previouslycalculated points of the plot 200 (the points that the plotter used togenerate the pre-updated version of the result surface) in combinationwith the next plot points just calculated. For example, the plotter 72may use a curve-fitting algorithm to generate a representation of theupdated result surface equal to a representation of a surface that bestfits this combination of plot points. The plotter 72 may also update thelocation of the reference point 214.

Next, the plotter 72 generates representations of the updated $100 Mlevel-set region 206 and updated level set 208 based on therepresentation of the updated result surface 202—because the resultsurface is updated, it may have a different shape that shown in FIGS.17-18.

Then, the determiner 64, simulator 66, calculator 70, and plotter 72repeat the above procedure until the iterations of the updated resultsurface 202 have sufficiently converged to a final result surface thatis sufficient to allow an operator to select the values of vaccinationcoverage and bed-net coverage that he/she feels are the best values foractual implementation. For example, the operator may visually monitorthe changes from updated result surface to updated result surface viathe GUI 80, and instruct, via the interface 74, the determiner 64,simulator 66, calculator 70, and plotter 72 to repeat the aboveprocedure until he/she feels that the result surface has sufficientlyconverged toward a final form. Or, an operator may configure theinterface 74 to monitor the changes from updated result surface toupdated result surface (e.g., to monitor the entropy of the resultsurface), and to halt the above procedure when the change from updatedresult surface to updated result surface is below an operator-selectedthreshold.

An operator may select the values of the vaccination coverage and thebed-net coverage for implementation in an actual malaria-eliminationcampaign from a point of the plot 200 that is on or off the resultsurface 202. For example, an operator may select as these values ofvaccination coverage and bed-net coverage the respective P₁ and P₂coordinates of a valley point 220 within the level-set region 206because this valley point has the lowest value of the SC coordinate, andthus has the lowest average cost of vaccinations and bed nets. Or, anoperator may select as these values of vaccination coverage and bed-netcoverage the respective P₁ and P₂ coordinates of a point that is not onthe result surface 202, but that is near to a point of the resultsurface within the level-set region 206 or on the level set 208.

Still referring to FIGS. 6, 17, and 18, alternate embodiments of theabove-described next-value-determination-and-result-surface-updatingtechnique are contemplated. For example, alternate embodiments describedabove in conjunction with FIGS. 14-16 may be applicable to the techniquedescribed above in conjunction with FIGS. 17-18. Moreover, an embodimentof the technique described above in conjunction with FIGS. 17-18 may beuseful for sought-after results other than a cost of a diseaseintervention being below a cost threshold.

FIG. 19 is a plot 230 of an N=3-dimensional result surface 232, whichincludes a valley 234; as described above, the plotter 72 (FIG. 6) maygenerate a representation of the result surface, and the interface 74(FIG. 6) may convert this representation into a form suitable fordisplaying the plot of the result surface via the GUI 80 (FIG. 6).

Referring to FIGS. 6 and 19, in an example that is similar to thatdescribed above in conjunction with FIGS. 17-18, assume that thesimulation tool 60 is simulating a malaria-transmission system, thedimensions of the result surface 232 are P₁=vaccination coverage,P₂=bed-net coverage, and SC=average cost of the vaccinations and bednets over the course of a malaria-elimination campaign, and an operatorof the simulation tool 60 is desirous of discovering what, if any,values of P₁ and P₂ yield a value of SC no higher than $100 M.

Therefore, the plotter 72 (FIG. 6) determines if there is a level set at$100 M that includes at least one point of the result surface 232.Conceptually, the plotter 72 may generate a representation of animaginary plane 236 at SC=$100 M that is parallel with the P₁-P₂ plane,and any portions of the result surface 232 that are level with thisplane form the $100 M level set.

But in this example, unlike the example described above in conjunctionwith FIGS. 17-18, there are no portions of the peak 234 of the resultsurface 232 that are level with or lower than the imaginary plane 236;therefore, there is no $100 M level set.

Referring to FIGS. 6 and 19, operation of an embodiment of thedeterminer 64, simulator 66, calculator 70, and plotter 72 is describedin conjunction with the result surface 232.

In an effort to determine whether a $100 M level set exists, thedeterminer 64 selects next values of vaccination coverage P₁ and bed-netcoverage P₂ for simulation in response to the peak 234. For example, thedeterminer 64 may randomly select a point 238 of the plot 230 that iswithin a particular radial distance R from a reference point 240 of thepeak 234, and the P₁ and P₂ coordinates of this randomly selected point238 are the next values of vaccination coverage and bed-net coverage,respectively. The particular values for the radial distance R and thereference point 240 of the result surface 232 from which the radialdistance R is measured may be selected by an operator of the tool 60 viathe GUI 80. For example, the radial distance R may be √{square root over(N_(t))} and the reference point 240 of the result surface 232 may bethe center of mass of the peak 234. It is noted that neither therandomly selected point 238 nor the reference point 240 need lie on theresult surface 232, although one or both points may lie on the resultsurface. Where the randomly selected point 238 lies on the resultsurface 232, a simulation using the P₁ and P₂ coordinates of therandomly selected point is typically not a “wasted” situation forreasons similar to those described above in conjunction with FIGS.14-15.

Consequently, instead of haphazardly selecting next values of thevaccination coverage and bed-net coverage, the determiner 64 selects thenext values for these parameters in a methodical manner that may reducethe overall number of simulation points that the simulation tool 60needs to simulate in order to determine whether there exist values ofthe vaccination coverage and bed-net coverage that yield thesought-after result of a not-more-than $100 M average cost forvaccinations and bed nets. That is, by selecting the next values ofvaccination coverage and bed-net coverage from nearby the peak 234, thedeterminer 64 may avoid a “wild goose chase” of next values that areunlikely to yield information as to whether there exist values ofvaccination coverage and bed-net coverage that yield an average costthat is $100 M or lower.

Still referring to FIGS. 6 and 19, the determiner 64 then passes thesenext values of vaccination coverage and bed-net coverage, along with thevalues of any other input data and input parameters that define a fullsimulation point, to the simulator 66, which runs a number ofsimulations sufficient to allow the calculator 70 to yield a value ofthe average cost having one or more uncertainties and standarddeviation/variance within respective thresholds, for example, asdescribed above in conjunction with FIG. 9.

Next, the plotter 72 generates a representation of a next point 242 ofthe plot 230 having coordinates (P₁, P₂, SC) respectively equal to thenext values of the vaccination coverage and bed-net coverage selected bythe determiner 64, and to the next value of the average cost calculatedby the calculator 70. Although the next point 242 and the point 238 havethe same P₁ and P₂ coordinates, they almost certainly have different SCcoordinates for reasons similar to those described above in conjunctionwith FIGS. 14-15.

Then, the interface 74 saves the representation of the next point 242 ofthe plot 230 in the output database 76; that is, the interface saves inthe output database the values of the input data and input parametersthat yielded the next point 242 as well as the SC coordinate of thisnext point.

Next, the determiner 64 selects next values of the vaccination coverage(P₁) and bed-net coverage (P₂) nearby the peak 234, and the simulator66, calculator 70, and plotter 72 repeat their respectiveabove-described operations, which results in the plotter generating arepresentation of a next point 244 of the plot 230.

The determiner 64, simulator 66, calculator 70, and plotter 72 continueto repeat their above-described operations until the plotter hasgenerated a number of next plot points. An operator of the tool 60 mayselect this number, which may be greater than or equal to one, via theinterface 74.

Then, the plotter 72 generates a representation of an updated resultsurface 232 using the previously calculated points of the plot 230 (thepoints that the plotter used to generate the pre-updated version of theresult surface) in combination with the next plot points justcalculated. For example, the plotter 72 may use a curve-fittingalgorithm to determine the updated result surface as a surface that bestfits this combination of plot points. And the plotter 72 may also updatethe location of the reference point 240.

Next, the plotter 72 evaluates the updated result surface 232 anddetermines whether a $100 M level set exists—the updated result surface232, because it is updated, may have a different shape than shown inFIG. 19.

If no $100 M level set exists, then the determiner 64, simulator 66,calculator 70, and plotter 72 repeat the above procedure until theiterations of the result surface 232 result in a $100 M level set, orhave converged toward a final result surface that is sufficient toindicate that no $100 M level set is likely to exist. For example, anoperator user may visually monitor the changes from updated resultsurface to updated result surface via the GUI 80, and instruct, via theinterface 74, the determiner 64, simulator 66, calculator 70, andplotter 72 to repeat the above procedure until a $100 M level setexists, or until he/she feels that the result surface as sufficientlyconverged toward a final form sufficient to indicate that a $100 M levelset is unlikely to exist. Or, an operator may configure the interface 74to monitor the changes from updated result surface to updated resultsurface (e.g., to monitor the entropy of the result surface), and tohalt the above procedure when either a $100 M level set is discovered orthe average change from updated result surface to updated result surfaceis below an operator-selected threshold.

If a $100 M level set is found, then the determiner 64, simulator 66,calculator 70, and plotter 72 may revert to an embodiment of theprocedure described above in conjunction with FIGS. 17-18.

If no $100 M level set is found, then an operator may “tweak” one ormore values of the input data and input parameters other thanvaccination coverage and bed-net coverage used to form the fullsimulation point, or the operator may take other action. For example, anoperator may decide to investigate disease-intervention parameters otherthan vaccination coverage and bed-net coverage in a manner similar tothat described above for the investigation of vaccination coverage andbed-net coverage.

Still referring to FIGS. 6 and 19, alternate embodiments of theabove-described next-value-determination-and-result-surface-updatingtechnique where no sought-after level set exists (at least initially)are contemplated. For example, alternate embodiments described above inconjunction with FIGS. 14-18 may also be compatible with an embodimentdescribed in conjunction with FIG. 19.

FIG. 20 is a plot 260 of an N=3-dimensional result surface 262, whichincludes a slope 264; as described above, the plotter 72 (FIG. 6) maygenerate a representation of the result surface, and the interface 74(FIG. 6) may convert this representation into a form suitable fordisplaying the plot of the result surface via the GUI 80 (FIG. 6).

FIG. 21 is a plan view of the plot 260, result surface 262, slope 264, alevel-set region 266 and a corresponding level set 268, a level-setregion 270 and a corresponding level set 272, and a slope region 273 anda corresponding slope-region boundary 274.

Referring to FIGS. 6, 20, and 21 and as further described below, thedeterminer 64 may determine a next value of an input parameter inresponse to the level sets 268 and 272 and the slope-region boundary 274when an operator of the simulation tool 60 is, for example, trying todetermine the sensitivity of a sought-after result to one or more inputparameters. That is, an operator may be trying to find regions of theresult surface 262 where the simulated system may be unstable. Asdiscussed above in conjunction with FIG. 9, an operator may do this todetermine the possible effect that parameter, coverage, or modeluncertainty may have on a simulation result.

For example, assume that the simulation tool 60 is simulating amalaria-transmission system, the dimensions of the result surface 262are P₁=vaccination coverage, P₂=bed-net coverage, and SC=the probabilityof successful elimination of malaria, and an operator of the simulationtool is desirous of discovering and identifying values of P₁ and P₂ towhich the value of SC is sensitive.

Along the slope 264 within the slope region 273, SC is sensitive tovalues of P₁ and P₂ because a relatively small increase in one or bothof P₁ and P₂ may cause a relatively large increase in the value of SC,and because a relatively small decrease in one or both of P₁ and P₂ maycause a relatively large decrease in the value of SC.

If, for example, the level-set region 270 represents simulation pointshaving a desired probability of success SC (e.g., 99.0% or greater),then an operator may wish to avoid selecting, for use in an actualmalaria-elimination campaign, values of P₁ and P₂ within the level-setregion 270 that are also near the slope regionset 273, because just aslight deviation in one of these values or in another parameter orsystem value (e.g., due to a simulation error, parameter, coverage, ormodel uncertainty, or other changes in the actual system compared to thesimulated system) may cause the campaign to “fall off a cliff.” That is,just a slight deviation in a campaign parameter (e.g., vaccinationcoverage, bed-net coverage) or in a system condition (e.g., averagetemperature, average rainfall) or other system quantity may reduce theprobability of successful elimination to well below the sought-afterlevel; or, to put it more succinctly, if one bases an eliminationcampaign on a simulation point that is within the level-set region 270but that is also near the slope region 273, then even a slight deviationin the actual values of any of the campaign parameters or systemconditions/quantities relative to the simulated values may mean thedifference between a successful result and an unsuccessful, or even adisastrous, result.

Therefore, an operator may configure the plotter 72 to identify a slopeof a result surface such as the slope 264, may identify a slope region,such as the slope region 273, that includes the slope and that is nearone or more level sets (e.g., the level set 272) of interest, and mayconfigure the determine 64 to select values of P₁ and P₂ near theidentified slope region so as to further define the slope region and itsslope-region boundary. For example, the plotter 72 may identify as aslope region a region of the result surface 262 having a slope magnitude(rise/run) in the SC dimension that is greater than a threshold (e.g.,25°); or the plotter may identify as a slope region a region of theresult surface having a slope magnitude in the SC dimension that isgreater than a first threshold (e.g., 25°), and may define theslope-region boundary of the slope region as the points of the resultsurface were the slope magnitude drops to a second threshold (e.g., onehalf of the first threshold). The latter slope-region identificationtechnique may impart an increased margin of stability for an actualcampaign point that is selected away from the slope region because sucha campaign point may be farther away from the steepest slope of theresult surface than if the former slope-region identification techniquewere used.

Referring to FIGS. 6, 20, and 21, operation of an embodiment of thedeterminer 64, simulator 66, calculator 70, and plotter 72 is describedin conjunction with the result surface 262, level-set regions 266 and270, level sets 268 and 272, slope region 273, and slope-region boundary274.

In an effort to further define the slope-region boundary 274 of theslope region 273, which includes the slope 264, the determiner 64selects next values of vaccination coverage P₁ and bed-net coverage P₂for simulation in response to the slope-region boundary 274. Forexample, the determiner 64 may randomly select a point 276 of the plot260 that is within a particular radial distance R from a reference point278 of the result surface 262, and the P₁ and P₂ coordinates of thisrandomly selected point 276 are the next values of vaccination coverageand bed-net coverage, respectively. The particular values for the radialdistance R and the reference point 278 of the result surface 262 fromwhich the radial distance R is measured may be selected by a user of thetool 60 via the GUI 80. For example, the radial distance R may be√{square root over (N_(t))} and the reference point 278 may be thecenter of mass of the slope region 273. It is noted that neither therandomly selected point 276 nor the reference point 278 need lie on theresult surface 262, although one or both of these points may lie on theresult surface. Where the randomly selected point 276 lies on the resultsurface 262, a simulation using the P₁ and P₂ coordinates of therandomly selected point is typically not a “wasted” situation forreasons similar to those described above in conjunction with FIGS.14-15.

Consequently, instead of haphazardly selecting next values of thevaccination coverage and bed-net coverage, the determiner 64 selects thenext values for these parameters in a methodical manner that may reducethe overall number of points that the tool 60 needs to simulate forsufficiently defining the slope-region boundary 274. That is, byselecting the next values of vaccination coverage and bed-net coveragefrom within or nearby the slope region 273 or from on or nearby theslope-region boundary 274, the determiner 64 may avoid a “wild goosechase” of next values that are unlikely to yield significant informationregarding the location of the slope-region boundary 274.

Still referring to FIGS. 6, 20, and 21, the determiner 64 then passesthese next values of vaccination coverage and bed-net coverage, alongwith the values of any other input data or input parameters that definea full simulation point, to the simulator 66, which runs a number ofsimulations sufficient to allow the calculator 70 to yield a value ofthe probability of successful malaria elimination having one or moreuncertainties and variance within respective thresholds, for example asdescribed above in conjunction with FIG. 9.

Next, the plotter 72 generates a representation of a next point 280 ofthe plot 260 having coordinates (P₁, P₂, SC) respectively equal to thenext values of the vaccination coverage and bed-net coverage selected bythe determiner 64, and to the next value of the probability of successcalculated by the calculator 70. Although the points 276 and 280 havethe same P₁ and P₂ coordinates, they almost certainly have different SCcoordinates as described above in conjunction with FIGS. 14-15.

Then, the interface 74 saves the representation of the next point 280 ofthe plot 260 in the output database 76; that is, the interface saves theP₁, P₂, and SC coordinates of the next point 280 as well as the valuesof the input data and other input parameters that yielded this nextpoint.

Next, the determiner 64 uses the slope-region boundary 274 to selectnext values of the vaccination coverage (P₁) and bed-net coverage (P₂),and the simulator 66, calculator 70, and plotter 72 repeat theirrespective above-described operations, which result in a representationof a next point 282 of the plot 260.

The determiner 64, simulator 66, calculator 70, and plotter 72 continueto repeat their above-described operations until the plotter hasgenerated a number of next plot points sufficient to update the resultsurface 262. An operator of the simulation tool 60 may select thisnumber, which may be greater than or equal to one, via the interface 74.

Then, the plotter 72 updates the result surface 262 using the previouslycalculated points of the plot 260 (the points that the plotter used togenerate the pre-updated version of the result surface) in combinationwith the next plot points (e.g., 280, 282) just calculated. For example,the plotter 72 may use a curve-fitting algorithm to determine theupdated result surface as a surface that best fits this combination ofplot points. The plotter 72 may also update the location of thereference point 278. The plotter 72 then generates a representation ofthe updated result surface 262; because the result surface is updated,it may have a different shape than appears in FIGS. 20 and 21.

Next, the plotter 72 updates the level sets 268 and 272 and theslope-region boundary 274 based on the updated result surface262—because of the updating, these updated level sets and slope-regionboundary may look different than they appear in FIGS. 20 and 21.

Then, the determiner 64, simulator 66, calculator 70, and plotter 72repeat the above procedure until the iterations of the result surface262 have sufficiently converged toward a final result surface with afinal slope region 273 and slope-region boundary 274. For example, anoperator of the simulation tool 60 may visually monitor the changes fromupdated result surface to updated result surface via the GUI 80, andinstruct, via the interface 74, the determiner 64, simulator 66,calculator 70, and plotter 72 to repeat the above procedure until he/shefeels that the result surface as sufficiently converged toward a finalform. Or, an operator may configure the interface 74 to monitor thechanges from updated result surface to updated result surface (e.g., tomonitor the entropy of the result surface), and to halt the aboveprocedure when the change from updated result surface to updated resultsurface is below an operator-selected threshold.

After the slope region 273 and slope-region boundary 274 aresufficiently defined, an operator may continue the simulation to findsuitable values for vaccination coverage P₁ and bed-net coverage P₂ thatcorrespond to a sought-after probability of success as described abovein conjunction with FIGS. 14-16 to further develop, for example, thelevel set 268.

Then, an operator may select the values of the vaccination coverage andthe bed-net coverage to be used in an actual disease-eliminationcampaign from a point of the plot 260 that is on or off the resultsurface 262 as described above in conjunction with FIGS. 14 and 15, withthe additional restriction that the point is a safe distance away fromthe slope-region boundary 274. That is, an operator may select thevalues of vaccination coverage and bed-net coverage to be used in anactual campaign from a point that corresponds to a margin of systemstability greater than a threshold that may be determined based on thedisease to be eliminated or on other criteria.

Still referring to FIGS. 6, 20, and 21, alternate embodiments of theabove-described next-value-determination-and-result-surface-updatingtechnique are contemplated. For example, alternate embodiments describedabove in conjunction with FIGS. 14-19 may be application to theprocedure described above in conjunction with FIGS. 20-21. Furthermore,although the slope-region boundary 274 is described as a beingcoincident with portions of the level sets 268 and 272, the boundary 274may be coincident with only one, or with none, of the level sets 268 and272.

FIG. 22 is a set of plots 300 ₁-300 _(y) of respective result surfaces302 ₁-302 _(y) each having at least one common dimension; for example,one or more of the result surfaces 302 ₁-302 _(y) may be similar to oneor more of the result surfaces 152, 182, 202, 232, and 262 of FIGS.14-21; as described above, the plotter 72 (FIG. 6) may generaterespective representations of the result surfaces 302 ₁-302 _(y), andthe interface 74 (FIG. 6) may convert these representations into a formssuitable for displaying the plots of the result surfaces via the GUI 80(FIG. 6).

Referring to FIGS. 6 and 22, the determiner 64 may select a respectivenext value of one or more input parameters based on more than one of theresult surfaces 302 ₁-302 _(y), for example, when an operator of thesimulation tool 60 is desirous of finding values of the inputparameter(s) that satisfy multiple criteria.

For example, suppose an operator wishes to determine whether there existvalues of vaccination coverage and bed-net coverage that yield both aprobability of successful malaria elimination that is 99% or greater atan average cost of $100 M or less.

He/she may configure the determiner 64 to use the plots 150 of FIGS.14-15 and 200 of FIGS. 17-18 to select next values of vaccinationcoverage and bed-net coverage in response to both of the level sets 158(FIGS. 14-15) and 208 (FIGS. 17-18). Specifically, the determiner 64 mayselect next values of vaccination coverage and bed-net coverage fromrespective simulation points that are within radii R₁ and R₂ of thereference points 164 and 214 of the plots 150 (FIGS. 14-15) and 200(FIGS. 17-18), respectively—R₁ and R₂ may, but need not be, equal to oneanother.

Then, the determiner 64, simulator 66, calculator 70, and plotter 72 mayoperate as described above in conjunction with FIGS. 14-15 and 17-18 todevelop the result surfaces 152 and 202 sufficiently for an operator toidentify values of vaccination coverage and bed-net coverage that yielda probability of elimination of at least 99% at an average cost of nomore than $100 M.

If the determiner 64 can find no next values of vaccination coverage andbed-net coverage that are within the specified ranges of both level sets158 and 208, then it may notify an operator via the GUI 80.

In response to such notification, an operator may configure thesimulation tool 60 to “relax” one or both of the next-point-selectioncriteria. For example, one may configure the determiner 64 to use a$150M level set in the plot 200 instead of a $100 M level set, or to usea 98% level set in the plot 150 instead of a 99% level set. Or, anoperator may shift the reference points 164 (FIGS. 14-15) and 214 (FIGS.17-18), e.g., closer to the respective level sets 158 and 208 to allowselection of next values farther outside of these level sets.

Or, an operator may configure the simulation tool 60 to automaticallyrelax one or both of the next-point-selection criteria. For example, anoperator may configure the tool 60 such that it steps the probabilitylevel set down from 99% to 98% in 0.1% intervals, steps the average-costlevel set up from $100 M to $150M in $5M intervals, or moves thereference points closer to the respective level sets in intervals of R/Nuntil the determiner 64 is able to find a simulation point that iswithin R₁ and R₂ of both reference points, respectively.

An operator may also use multiple ones of the plots 300 ₁-300 _(y) todetermine which values of multiple input parameters yield a samesought-after result. For example, suppose an operator wants to determinewhich set of values of vaccination coverage, bed-net coverage,mosquitocide coverage, and malaria-prevention-education coverage yield aprobability of eliminating malaria greater than or equal to 99%. Insteadof configuring the plotter 72 (FIG. 6) to generate a representation ofan N=5-dimensional plot, to reduce processing complexity and tofacilitate plot display, an operator may configure the plotter togenerate two N=3-dimensional plots having the following sets ofdimensions: 1) P_(1,1)=vaccination coverage, P_(2,1)=bed-net coverage,SC₁=probability of elimination, and 2) P_(1,2)=mosquitocide coverage,P_(2,2)=education coverage, and SC₂=probability of elimination.Therefore, in this example, these two plots have only their SCdimensions in common.

Still referring to FIG. 22, alternate embodiments of the plots 300 ₁-300_(y) and procedures for using these plots to select one or more nextinput-parameter values are contemplated. For example, one or more of thealternate embodiments described above in conjunction with FIGS. 14-21may be applicable to the plots 300 ₁-300 _(y) and the procedures forusing these plots. Furthermore, although all shown having the samenumber of dimensions, these plots may have different numbers ofdimensions.

FIG. 23 is a flow chart 310 of an embodiment of a system-simulationprocedure that an embodiment of the simulation tool 60 of FIG. 6 mayimplement.

Referring to FIG. 6 and to a block 312 of FIG. 23, the tool 60 starts asimulation of a stochastically modeled system.

Next, referring to block 314 of FIG. 23, the simulator 66 propagates atleast one state of the simulated system through time in response to avalue of an input parameter received from the determiner 64. Forexample, if the system model 68 is model of a malaria-transmissionsystem, then the simulator 66 may propagate through time a state thatrepresents the number of mosquitoes infected with malaria in response toone or more input-data and/or input-parameter values such as a value ofvaccination coverage received from the determiner 64.

Then, referring to block 316, the determiner 64 determines a next valueof the input parameter in response to a characteristic of a state of themodel and a representation of at least one level set. For example,referring to FIGS. 14-15, the determiner 64 may determine a next valuefor vaccination coverage P₁ in response to the probability of successfuleradication SC, which is a characteristic of a state that represents thenumber of malaria-infected individuals, and at least one level set 158.In this example, the points of the plot 150 from which the determiner 64may determine the next value of vaccination coverage P₁ are within aradius R of a level-set reference point 164; therefore, these points arewithin a range of SC values dictated by the level set 158. That is, thedeterminer 64 determines a next value of vaccination coverage P₁ byselecting a point (e.g., point 166) of the plot 150 that is within asphere of radius R from the level-set reference point 164; consequently,the only points that the determiner may select are in a spherical regionof the plot 150 where the probability of success SC is within a certainrange of values.

Still referring to FIGS. 6 and 23, alternate embodiments of thesimulation procedure are contemplated. For example, alternateembodiments described above in conjunction with FIGS. 6-22 may beapplicable to the simulation procedure of FIG. 23. Furthermore, althoughdescribed as being different states, the state that the simulator 66propagates through time may be the same as the state having the statecharacteristic. For example, the simulator 66 may propagate a staterepresenting the number of infected individuals through time, and thestate characteristic may be the probability of successful eradication,which is a characteristic of the state representing the number ofinfected individuals. Moreover, although described as determining a nextvalue for only one parameter, an embodiment of the procedure maydetermine respective next values for multiple parameters. Moreover,although described as being responsive to a state characteristic and alevel set, an embodiment of the procedure may be responsive to a statecharacteristic and multiple level sets, to multiple statecharacteristics and a level set, or to multiple state characteristicsand multiple level sets. In addition, any step described as beingperformed by a component of the simulation tool 60 may be performed byanother component of the tool, or by any combination of components ofthe tool. In addition, although described as a stochastically modeledsystem, the system may be modeled other than stochastically.Furthermore, although described as propagating a state of the simulatedsystem through time, the simulator 66 (FIG. 6) may simulate the systemin another manner.

FIG. 24 is a flow chart 320 of an embodiment of a simulation procedurethat an embodiment of the simulation tool 60 of FIG. 6 may implement.

Referring to FIG. 6 and to block 322 of FIG. 24, the tool 60 starts asimulation of a stochastically modeled system.

Next, referring to block 324 of FIG. 24, the simulator 66 propagates atleast one state of the simulated system through time in response to avalue of an input parameter received from the determiner 64. Forexample, if the system model 68 is model of a malaria-transmissionsystem, then the simulator 66 may propagate through time a state thatrepresents the number of mosquitoes infected with malaria in response toone or more input-data or input-parameter values, such as a value ofvaccination coverage received, from the determiner 64.

Then, referring to block 326, the plotter 72 generates a representationof a region of a plot having dimensions that respectively correspond atleast to the parameter and to a characteristic of a state of thesimulated system. For example, referring to FIGS. 14-15, the plotter 72may generate the result surface 152 (a region) of the plot 150, and theresult surface has dimensions P₁ and SC that respectively correspond tothe parameter such as vaccination coverage and to a probability ofsuccessful elimination, which is a characteristic of thesimulated-system state representing the number of infected individuals.

Next, referring to block 328, the determiner 64 determines a next valueof the parameter in response to the representation of the region of theplot. For example, referring to FIGS. 14-15, the determiner 64 maydetermine a next value of the input parameter vaccination coverage inresponse to a representation of the result surface 152 of the plot 150.More specifically, the determiner 64 may determine a next value of theinput parameter vaccination coverage in response to the representationof the portion of the result surface 152 within the level-set region156.

Still referring to FIGS. 6 and 24, alternate embodiments of thesimulation procedure are contemplated. For example, alternateembodiments described above in conjunction with FIGS. 6-23 may beapplicable to the simulation procedure of FIG. 23.

FIG. 25 is a flow chart 340 of an embodiment of a simulation procedurethat an embodiment of the simulation tool 60 of FIG. 6 may implement.

Referring to FIG. 6 and to a block 342 of FIG. 25, the tool 60 starts asimulation of a stochastically modeled system.

Next, referring to block 344 of FIG. 25, the simulator 66 propagates atleast one state of the simulated system through time in response to avalue of an input parameter received from the determiner 64. Forexample, if the system model 68 is model of a malaria-transmissionsystem, then the simulator 66 may propagate through time a state thatrepresents the number of mosquitoes infected with malaria in response toone or more input-data or input-parameter values such as a value ofvaccination coverage received from the determiner 64.

Then, referring to block 346, the plotter 72 generates a representationof a region of a first plot having dimensions that respectivelycorrespond at least to the parameter and to a characteristic of a stateof the simulated system. For example, referring to FIGS. 14-15, theplotter 72 may generate the result surface 152 (a region) of the plot150, and the result surface has dimensions P₁ and SC that respectivelycorrespond to the parameter such as vaccination coverage and to aprobability of successful elimination, which is a characteristic of thesimulated-system state representing the number of infected individuals.

Then, referring to block 348, the plotter 72 generates a representationof a region of a second plot having dimensions that respectivelycorrespond at least to the parameter and to a characteristic of a stateof the simulated system. For example, referring to FIGS. 17-18, theplotter 72 may generate the result surface 202 (a region) of the plot200, and the result surface has dimensions P₁ and SC that respectivelycorrespond to the parameter such as vaccination coverage and to anaverage cost of vaccination coverage and bed-net coverage, which is acharacteristic of the simulated-system state representing the cumulativecost of vaccination coverage and bed-net coverage.

Next, referring to block 350, the determiner 64 determines a next valueof the parameter in response to the representations of the regions ofthe first and second plots. For example, referring to FIGS. 14-15 and17-18, the determiner 64 may determine a next value of the inputparameter vaccination coverage in response to representations of theresult surfaces 152 and 202 of the plots 150 and 200, respectively. Morespecifically, the determiner 64 may determine a next value of the inputparameter vaccination coverage in response to the representations of theportions of the result surfaces 152 and 202 within the level-set regions156 and 206, respectively, in an effort to find a value of vaccinationcoverage that corresponds to an elimination probability of 99% orgreater and an average cost of $100 M or less.

Still referring to FIGS. 6 and 25, alternate embodiments of thesimulation procedure are contemplated. For example, alternateembodiments described above in conjunction with FIGS. 6-24 may beapplicable to the simulation procedure of FIG. 25. Furthermore, althoughdescribed as being different, two or more of the described states may bethe same, and the described state characteristics may be the same.Moreover, although described as using the regions from two plots, theprocedure may be responsive to regions from more than two plots, andthese plots may collectively include as dimensions more than two statecharacteristics and more than two input parameters.

FIG. 26 is a flow chart 360 of an embodiment of a procedure that anembodiment of the simulation tool 60 of FIG. 6 may implement to generaterespective representations of one or more initial result surfaces.

Referring to FIG. 6 and to a block 362 of the FIG. 26, the tool 60starts a simulation of a stochastically modeled system.

Next, referring to block 364 of FIG. 26, an operator of the tool 60selects, via the GUI 80, the input data and the input parameters thatwill collectively form a simulation point. For example, if an operatorwishes to model a malaria-transmission system in Madagascar, then he/shemay select input-data categories (e.g., average temperature, averagerainfall, population) that the input data base 62 stores for Madagascar,and may select input parameters specific to malaria elimination (e.g.,vaccination coverage, bed-net coverage, mosquitocide coverage).

Then, referring to block 366, an operator selects, via the GUI 80, thevalues for any input data (e.g., weather-related input data) or inputparameters that are to remain constant or otherwise unchanged from theirrespective initial values during the simulation. For example, for theinput data, an operator may select values that are stored in the inputdatabase 62.

Next, referring to block 368, the determiner 64 determines initialvalues of the input parameters to be “tuned,” i.e., the input parametersfor which the determiner will attempt to find values that correspond toa sought-after result. For example, for a malaria-transmission system,an operator may configure the determiner 64 to find values ofvaccination coverage and bed-net coverage that correspond to at least a99% probability of successful elimination. For example, the determiner64 may determine the initial values of the input parameters to be tunedaccording to an embodiment described above in conjunction with FIGS. 12and 13.

Then, referring to block 370, the simulator 66 simulates thestochastically modeled system over the period of interest (e.g., adisease-elimination campaign) in response to the input-data values andthe input-parameter values (including the initial values for the inputparameters to be tuned) received from the determiner 64.

Next, referring to block 372, the interface 74 saves the simulationresults (e.g., a snapshot of the states of the simulated system at eachtime step) in the output database 76.

Then, referring to block 374, the calculator 70 increments a value A,which has an initial value of zero and which represents the number ofsimulation runs that the simulator 66 has performed at the currentsimulation point.

Next, referring to block 376, the calculator 70 determiners if A isgreater than or equal to a threshold number of simulation runs. In anembodiment, the threshold number is a number sufficient for thecalculator 70 to yield a state characteristic (e.g., probability ofsuccessful elimination) having one or more uncertainties and standarddeviation/variance within respective thresholds. The threshold number ofsimulation runs may be selected by an operator of the simulation tool60. Or, the calculator 70 may effectively calculate the threshold numberby calculating the uncertainty(ies) and standard deviation/variance ofthe result after each simulation run, and by effectively indicating thatA is greater than or equal to the threshold number when theuncertainty(ies) and standard deviation/variance of the result arewithin respective thresholds.

If A is not actually or effectively greater than or equal to thethreshold number of runs through the simulator 66, then the procedurereturns to block 370, and repeats the steps represented by the blocks370, 372, 374, and 376 until A is actually or effectively greater thanthe threshold number of simulation runs.

Then, referring to block 378, the calculator 70 calculates acharacteristic (e.g., probability of successful elimination) of a stateof interest (e.g., the number of infected individuals at the end of asimulated disease-elimination campaign period), where the calculatedcharacteristic has one or more uncertainties and standarddeviation/variance within respective thresholds.

Next, referring to block 380, the plotter 72 generates a representationof a plot point in response to the calculated characteristic (e.g.,probability of successful elimination) and the to-be-tuned inputparameters (e.g., vaccination coverage and bed-net coverage). In anembodiment, the plotter 72 generates the plot point having coordinatesequal to the values of the calculated characteristic and the values ofthe to-be-tuned input parameters.

Then, referring to block 382, the plotter 72 increments a value B, whichis initially zero and which represents the number of plot pointsgenerated by the plotter.

Next, referring to block 384, the plotter 72 determines whether B isgreater than or equal to a threshold number of points that the plotterneeds to generate an initial version of a result surface. In anembodiment, the threshold number may be selected by an operator of thesimulation tool 60. Or, the plotter 72 may effectively calculate thethreshold number on a case-by-case basis depending on the number andlocations of the points, and by effectively indicating that B is greaterthan or equal to the threshold number when the number of points issufficient for generating an initial result surface.

If B is not greater than or equal to the threshold number of points,then, referring to block 386, the determiner 64 determines next valuesof the to-be-tuned input parameters, and the procedure repeats the stepsrepresented by the blocks 370, 372, 374, 376, 378, 380, 382, 384, and386 until B is greater than or equal to the threshold number of points.The determiner 64 may determine the next values of the to-be-tuned inputparameters according to an embodiment described above in conjunctionwith FIGS. 12-13.

Referring again to block 384, if B is greater than or equal to thethreshold number of points, then, referring to block 388, the plotter 72generates a representation of an initial result surface from thegenerated plot points. In an embodiment, the plotter 72 generates arepresentation of the initial result surface by applying a conventionalcurve-fitting algorithm to the plot points.

Still referring to FIGS. 6 and 26, alternate embodiments of theprocedure are contemplated. Embodiments described above in conjunctionwith FIGS. 12-25 may be applicable to an embodiment of the procedure ofFIG. 26. Furthermore, although described as calculating a single statecharacteristic for a single state, an embodiment of the procedure maycalculate multiple state characteristics for a single state, a singlestate characteristic for multiple states, or multiple statecharacteristics for multiple states. Moreover, although described asgenerating a representation of an initial result surface, an embodimentof the procedure may generate respective representations of multipleinitial result surfaces; for example, the simulator tool 60 may repeatthe procedure represented by the flow chart 360 multiple times eitherconsecutively or concurrently, one time per each initial result surfaceto be generated.

FIG. 27 is a flow chart 400 of an embodiment of a procedure that anembodiment of the simulation tool 60 of FIG. 6 may implement to find avalue of an input parameter that corresponds to a sought-after result,or to discover that no such value exists.

Referring to FIG. 6 and to block 402 of FIG. 27, the tool 60 startsafter the plotter 72 has generated representations of one or moreinitial result surfaces, for example, as described above in conjunctionwith FIG. 26.

Next, referring to block 404, the plotter 72 generates a representationof a respective at least one level set for each initial result surfacerepresented. For example, the plotter 72 may generate such a level setaccording to an embodiment described above in conjunction with FIGS.14-22. An operator of the tool 60 may define these level sets via theGUI 80. For example, for simulating a malaria transmission system, anoperator may define a level set for probability of successfulelimination equal to 99%.

Then, referring to block 406, the determiner 64 attempts to determine arespective next value for each of the to-be-tuned input parameters inresponse to at least one region of at least one result surface and inresponse to a respective at least one level set for each of these resultsurfaces. For example, for a malaria-transmission system, the determiner64 may select a next value of vaccination coverage from a plot pointthat is within a radius R of a reference point corresponding to a99%-success level set according to an embodiment described above inconjunction with FIGS. 14-15.

Next, referring to block 408, if the determiner 64 does not find nextvalues for all of the to-be-tuned input parameters, then the procedureproceeds to block 410.

Then, referring to block 410, the tool 60 “decides” whether thedeterminer 64 is to retry determining a next value of the parameter forwhich it could not find a next value. For example, the tool 60 maynotify an operator, who may decide to relax the next-value-determinationcriteria according to an embodiment described above in conjunction withFIGS. 16 and 19. Or, an operator may have previously configured the tool60 to automatically relax the next-value-determination criteriaaccording to an embodiment described above in conjunction with FIGS. 16and 19.

If the tool 60 or operator decides to retry the next-valuedetermination, then the procedure returns to block 406.

If, however, the tool 60 or operator decides not to retry the next-valuedetermination, for example, because a threshold number of retries hasalready been made, then the tool notifies the operator of the problemvia the GUI 80 and finishes at a block 432.

Referring again to block 408, if the determiner 64 is able to find nextvalues for all of the to-be-tuned parameters, then the simulation tool60 proceeds to the step represented by a block 412.

Next, referring to block 412, the simulator 66 simulates thestochastically modeled system over the period of interest (e.g., adisease-elimination campaign) in response to the input-data values andthe input-parameter values (including the determined next values for theinput parameters to be tuned) received from the determiner 64.

Then, referring to block 414, the interface 74 saves the simulationresults (e.g., a snapshot of the states of the simualted system at eachtime step) in the output database 76.

Next, referring to block 416, the calculator 70 increments a value C,which has an initial value of zero and which represents the number ofsimulation runs that the simulator 66 has performed at the currentsimulation point.

Then, referring to block 418, the calculator 70 determiners if C isgreater than or equal to a threshold number of simulation runs. In anembodiment, the threshold number is a number sufficient for thecalculator 70 to yield a state characteristic (e.g., probability ofsuccessful elimination) having one or more uncertainties and standarddeviation/variance within respective thresholds. The threshold number ofsimulation runs may be selected by an operator of the simulation tool60. Or, the calculator 70 may effectively calculate the threshold numberby calculating the uncertainty(ies) and standard deviation/variance ofthe result after each simulation run, and by effectively indicating thatC is greater than or equal to the threshold number of simulation runswhen the uncertainty(ies) and standard deviation/variance of the resultare within respective thresholds.

If C is not actually or effectively greater than or equal to thethreshold number of runs through the simulator 66, then the procedurereturns to block 412, and repeats the steps represented by the blocks412, 414, 416, and 418 until C is actually or effectively greater thanthe threshold number of simulation runs.

Next, referring to block 420, the calculator 70 calculates a respectivecharacteristic (e.g., probability of successful elimination) for each ofat least one state of interest (e.g., the number of infected individualsremaining at the end of a simulated campaign period), where thecalculated characteristic has one or more uncertainties and standarddeviation/variance within respective thresholds.

Then, referring to block 422, the plotter 72 generates a representationof a respective plot point for each result surface in response to thecorresponding calculated characteristic (e.g., probability of successfulelimination) and the corresponding to-be-tuned input parameters (e.g.,vaccination coverage and bed-net coverage). In an embodiment, theplotter 72 generates the representation of each plot point havingcoordinates equal to the values of the corresponding at least onecalculated characteristic and the values of the at least one to-be-tunedinput parameters.

Next, referring to block 424, the plotter 72 increments a value D, whichis initially zero and which represents the number of plot points thatthe plotter has generated for each result-surface plot. Alternatively,the plotter 72 may increment a respective value D for each resultsurface.

Then, referring to block 426, the plotter 72 determines whether D isgreater than or equal to a threshold number of points. In an embodiment,the threshold number of points that the plotter 72 generates forproducing an updated version of a result surface may be selected by anoperator of the simulation tool 60. Or, the plotter 72 may effectivelycalculate the threshold number of plot points on a case-by-case basisdepending on the number and locations of the points within a plot, andby effectively indicating that D is greater than or equal to thethreshold number when the number of points is sufficient for generatingan updated result surface. If the threshold number is different for eachof multiple result surfaces, then the plotter 72 may use as D thethreshold number of the result surface needing the greatest number ofpoints for updating, or the plotter may generate separate D values foreach result surface per above, and may indicate that the thresholdnumber of points has been reached for all result surfaces when each ofthese D values is greater than or equal to the number of points neededfor updating the corresponding result surface.

If D is not greater than or equal to the threshold number of points,then the simulation tool 60 returns to the step represented by the block406, and repeats the steps represented by the blocks 406, 408, 410 (ifapplicable), 412, 414, 416, 418, 420, 422, 424, and 426 until D isgreater than or equal to the threshold number of points.

Referring again to block 426, if D is greater than or equal to thethreshold number of points, then, referring to block 428, the plotter 72updates the representations of the result surfaces from the previouslygenerated plot points (used for generation of the initial result surfaceand any prior updates of the result surface) and just-generated plotpoints. In an embodiment, the plotter 72 updates the representation ofeach result surface by applying respective curve-fitting algorithms tothe respective sets of plot points.

Next, referring to block 430, the tool 60 determines whether the resultsurfaces are sufficiently developed to yield final results. For example,as described above in conjunction with FIGS. 14-22, either an operatoror the tool 60 may monitor the changes (e.g., the entropy) in the resultsurfaces from update to update, and determine that the surfaces aresufficiently developed if the level of change is less than or equal to athreshold level of change. The monitoring algorithm may be the same ordifferent for each result surface, and the operator/tool 60 may find onesurface sufficiently developed before it finds another surfacesufficiently developed.

If at least one of the result surfaces is not sufficiently developed toyield a final result, then the simulation tool 60 returns to the steprepresented by the block 406, and repeats the steps represented by theblocks 406, 408, 410 (if applicable), 412, 414, 416, 418, 420, 422, 424,426, 428, and 430 until all of the result surfaces are sufficientlydeveloped, or until the tool 60 times out because at least one of theresult surfaces is not converging toward a final surface. The tool 60may repeat these steps only for the result surfaces that are notsufficiently developed.

But if all result surfaces are sufficiently developed to yield a finalresult, then the tool 60 ends the procedure at block 432, and anoperator may analyze the plots of the result surfaces and level sets viathe GUI 80 (or in some other manner) and select values of the now-tunedinput parameters for use in an actual system.

Still referring to FIGS. 6 and 27, alternate embodiments of theprocedure are contemplated. Embodiments described above in conjunctionwith FIGS. 12-26 may be applicable to an embodiment of the procedure ofFIG. 27. Furthermore, an embodiment of the procedure may be applicableto any number of state characteristics and any number of to-be-tunedinput parameters per result-surface plot.

From the foregoing it will be appreciated that, although specificembodiments have been described herein for purposes of illustration,various modifications may be made without deviating from the spirit andscope of the disclosure. Furthermore, where an alternative is disclosedfor a particular embodiment, this alternative may also apply to otherembodiments even if not specifically stated.

While various aspects and embodiments have been disclosed herein, otheraspects and embodiments will be apparent to those skilled in the art.The various aspects and embodiments disclosed herein are for purposes ofillustration and are not intended to be limiting, with the true scopeand spirit being indicated by the following claims.

1. An apparatus, comprising: a simulator configured to simulated asystem and to propagate at least one state of the simulated systemthrough time in response to a value of a parameter; and a determinerconfigured to determine a next value of the parameter in response to acharacteristic of another state of the simulated system and arepresentation of at least one level set.
 2. The apparatus of claim 1wherein the representation of the at least one level set classifies thecharacteristic.
 3. The apparatus of claim 1 wherein the representationof the at least one level set classifies the characteristic byidentifying a region of a representation of a plot of thecharacteristic.
 4. The apparatus of claim 1 wherein the representationof the at least one level set classifies the characteristic byidentifying a change in a regime of the system related to the otherstate.
 5. The apparatus of claim 4 wherein the identified change in theregime includes a transition of the other state from having a tendencyto stay within approximately one range to having a tendency to movetoward approximately another range.
 6. The apparatus of claim 4 wherein:the other state relates to a disease; and the identified change in theregime includes a transition from the disease tending to persist to thedisease tending to fade out.
 7. The apparatus of claim 1 wherein thesimulator is configured to propagate the at least one state of thesimulated system through time by randomly generating a value of the atleast one state.
 8. The apparatus of claim 1 wherein the simulator isconfigured to propagate the at least one state of the simulated systemthrough time by generating a value of the at least one state in responseto a probability distribution of possible values of the at least onestate. 9-12. (canceled)
 13. The apparatus of claim 1 wherein: thesimulator is configured to simulate the system in response to a model ofthe system; and the model includes a model of a malady transmissionsystem.
 14. The apparatus of claim 13 wherein the parameter includes aparameter of the model.
 15. The apparatus of claim 13 wherein theparameter includes a malady intervention parameter.
 16. The apparatus ofclaim 1 wherein: the simulator is configured to propagate the at leastone state of the simulated system through time in response to a model ofan environment of the system; and the parameter includes a parameter ofthe model of the environment.
 17. The apparatus of claim 1 wherein: thesimulator is configured to propagate the at least one state of thesimulated system through time in response to a value of anotherparameter; and wherein the simulator is configured to modify the valueof the other parameter in response the value of the parameter.
 18. Theapparatus of claim 1 wherein: the simulator is configured to propagatethe at least one state of the simulated system through time in responseto a value of another parameter; and wherein the simulator is configuredto determine the value of the other parameter in response the value ofthe parameter.
 19. The apparatus of claim 1 wherein the determiner isconfigured to determine the next value of the parameter based on arelationship between a value of the characteristic of the other state ofthe simulated system and a value of the representation of the level set.20. The apparatus of claim 1 wherein the simulator is configured topropagate the at least one state of the simulated system through timefor a number of simulation runs each having a respective period.
 21. Theapparatus of claim 1 wherein the simulator is configured to propagatethe at least one state of the simulated system through time for a numberof simulation runs each having a same period.
 22. The apparatus of claim1 wherein: the simulator is configured to propagate the at least onestate of the simulated system through time for a number of simulationruns each having a respective period; and the characteristic of theother state includes a probability that a value of the other state iswithin a particular range.
 23. (canceled)
 24. The apparatus of claim 1wherein: the simulator is configured to propagate the at least one stateof the simulated system through time for a number of simulation runseach having a respective period; and the characteristic of the otherstate includes an uncertainty that a value of the other state is withina particular range.
 25. The apparatus of claim 1, further including:wherein the simulator is configured to propagate the at least one stateof the simulated system through time for a number of simulation runseach having a respective period, and to generate a respective value forthe other state after each of the simulation runs; and a calculatorconfigured to calculate from the respective values of the other state avalue of the characteristic of the other state.
 26. The apparatus ofclaim 1, further including: wherein the simulator is configured topropagate the at least one state of the simulated system through timefor a number of simulation runs each having a respective period, and togenerate a respective value for the other state after each of thesimulation runs; and a calculator configured to calculate a Bayesianprior of the respective values of the other state, and to calculate avalue of the characteristic of the other state from the Bayesian prior.27. The apparatus of claim 1, further including: wherein the simulatoris configured to propagate the at least one state of the simulatedsystem through time for a number of simulation runs each having arespective period, and to generate a respective value for the otherstate after each of the simulation runs; and a calculator configured tocalculate a beta function of the respective values of the other state,and to calculate a value of the characteristic of the other state fromthe beta function.
 28. The apparatus of claim 1, further including:wherein the simulator is configured to propagate the at least one stateof the simulated system through time for a number of simulation runseach having a respective period, and to generate a respective value forthe other state after each of the simulation runs; wherein thecharacteristic of the other state includes a probability that a value ofthe other state is within a particular range; and a calculatorconfigured to calculate from the respective values of the other state avalue of the probability.
 29. The apparatus of claim 1, furtherincluding: wherein the simulator is configured to propagate the at leastone state of the simulated system through time for a number ofsimulation runs each having a respective period, and to generate arespective value for the other state after each of the simulation runs;wherein the characteristic of the other state includes a probabilitythat a value of the other state is within a particular range; and acalculator configured to calculate a Bayesian prior of the respectivevalues of the other state, and to calculate a value of the probabilityfrom the Bayesian prior.
 30. The apparatus of claim 1, furtherincluding: wherein the simulator is configured to propagate the at leastone state of the simulated system through time for a number ofsimulation runs each having a respective period, and to generate arespective value for the other state after each of the simulation runs;wherein the characteristic of the other state includes a probabilitythat a value of the other state is within a particular range; and acalculator configured to calculate a beta function of the respectivevalues of the other state, and to calculate a value of the probabilityfrom the beta function.
 31. The apparatus of claim 1, further including:wherein the simulator is configured to propagate the at least one stateof the simulated system through time for a number of simulation runseach having a respective period, and to generate a respective value forthe other state after each of the simulation runs; wherein thecharacteristic of the other state includes an uncertainty that a valueof the other state is within a particular range; and a calculatorconfigured to calculate from the respective values of the other state avalue of the uncertainty.
 32. The apparatus of claim 1, furtherincluding: wherein the simulator is configured to propagate the at leastone state of the simulated system through time for a number ofsimulation runs each having a respective period, and to generate arespective value for the other state after each of the simulation runs;wherein the characteristic of the other state includes an uncertaintythat a value of the other state is within a particular range; and acalculator configured to calculate a Bayesian prior of the respectivevalues of the other state, and to calculate a value of the uncertaintyfrom the Bayesian prior.
 33. The apparatus of claim 1, furtherincluding: wherein the simulator is configured to propagate the at leastone state of the simulated system through time for a number ofsimulation runs each having a respective period, and to generate arespective value for the other state after each of the simulation runs;wherein the characteristic of the other state includes an uncertaintythat a value of the other state is within a particular range; and acalculator configured to calculate a beta function of the respectivevalues of the other state, and to calculate a value of the uncertaintyfrom the beta function.
 34. The apparatus of claim 1, further including:wherein the simulator is configured to propagate the at least one stateof the simulated system through time for a number of simulation runseach having a respective period, and to generate a respective value forthe other state after each of the simulation runs; a calculatorconfigured to determine from the respective values of the other state avalue of the characteristic of the other state; and a plotter configuredto generate a representation of a plot having coordinate dimensionsrespectively corresponding to at least the parameter and to thecharacteristic of the other state, a representation of a plot pointlocated on the plot at a coordinate corresponding to the value of theparameter and to the determined value of the characteristic of the otherstate, and the representation of the at least one level set located onthe plot at a coordinate corresponding to a level set value of thecharacteristic of the other state.
 35. The apparatus of claim 1, furtherincluding: wherein the simulator is configured to propagate the at leastone state of the simulated system through time for a number ofsimulation runs each having a respective period, and to generate arespective value for the other state after each of the simulation runs;a calculator configured to calculate from the respective values of theother state a value of the characteristic of the other state; a plotterconfigured to generate a representation of a plot having coordinatedimensions respectively corresponding to at least the parameter and thecharacteristic of the other state, a representation of a plot pointlocated on the plot at a coordinate corresponding to the value of theparameter and to the determined value of the characteristic of the otherstate, and the representation of the at least one level set located onthe plot at a coordinate corresponding to a level set value of thecharacteristic of the other state; and wherein the determiner isconfigured to determine the next value of the parameter in response tothe location of the representation of the plot point relative to therepresentation of the at least one level set.
 36. The apparatus of claim1, further including: a model of the system; and wherein the simulatoris configured to propagate the at least one state of the simulatedsystem through time in response to the model. 37-135. (canceled)